Contents
What is the best fitting model?
Analysis of the NLSY data Using the lowest BIC as a selection criterion, the best-fitting model was model 16. This model is a 2-class model with random intercepts and slopes, and it includes regression of the intercept factor on the three covariates in addition to the direct effect of class on the covariates.
Which of the following criteria is the best guide when selecting a suitable regression model?
R-Square values, Mallows Cp, Mean Square Error and minimum number of independent variables giving good/significant prediction of dependent variable are the good criteria to select the best model.
Which is the best strategy for model selection?
The recommended strategy for model selection depends on the amount of data available. If plenty of data is available, we may split the data into several parts, each serving a special purpose. For instance, for hyperparameter tuning we may split the data into three sets: train / validation / test.
How is validation set used in model selection?
The training set is used to fit the models; the validation set is used to estimate prediction error for model selection; the test set is used for assessment of the generalization error of the final chosen model. — Page 222, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2017.
How is the training set used in model selection?
The training set is used to train as many models as there are different combinations of model hyperparameters. These models are then evaluated on the validation set, and the model with the best performance on this validation set is selected as the winning model.
Do you need independent data for model selection?
To avoid such issues, we need completely independent data for estimating the generalization error of a model. We will come back to this point in the context of cross validation. The recommended strategy for model selection depends on the amount of data available.