Is there bias in feature selection in cross validation?

Is there bias in feature selection in cross validation?

Edit: On implementing feature selection within cross validation on the data set detailed above (thanks to the answers below), I can confirm that selecting features prior to cross-validation in this data set introduced a significant bias. This bias/overfitting was greatest when doing so for a 3-class formulation, compared to as 2-class formulation.

How is parameter estimation done in cross validation?

If the data-driven pre-processing is of type 1., this leads to “double” or “nested” cross validation: the parameter estimation is done in a cross validation using only the training set of the “outer” cross validation. The ElemStatLearn have an illustration ( https://web.stanford.edu/~hastie/Papers/ESLII.pdf Page 222 of print 5).

How does supervised feature selection lead to overfitting?

The suggestion is that any supervised feature selection (using correlation with class labels) performed outside of the model performance estimation using cross validation (or other model estimating method such as bootstrapping) may result in overfitting.

What is the error rate of cross validation?

If you perform feature selection independently within each fold of the cross-validation, the expected value of the error rate is 0.5 (which is correct). The key idea is that cross-validation is a way of estimating the generalisation performance of a process for building a model, so you need to repeat the whole process in each fold.

Do you need to cross validate a correlation?

Those are actually part of the model development and should be cross validated. What most people do is look at the correlations, select only the most correlated with the output, and then move on to do cross validation etc. That’s wrong for two reasons. Correlation measures strength of a linear relationship.

Which is better correlation or lasso for feature selection?

Correlation is myopic for the reasons I’ve described. Lasso is better, but there is no reason to think the features it selects are the “best” features, nor is there a reason to think that the features it selects would be selected had you had different data. Here is a code example to demonstrate that

When to perform feature selection before or after training?

This means that feature selection is performed on the prepared fold right before the model is trained. A mistake would be to perform feature selection first to prepare your data, then perform model selection and training on the selected features.

Which is the best method for cross validation?

K-Folds Cross Validation: K-Folds technique is a popular and easy to understand, it generally results in a less biased model compare to other methods. Because it ensures that every observation from the original dataset has the chance of appearing in training and test set. This is one among the best approach if we have a limited input data.

Can you run k-fold cross validation more than once?

Once model is selected again use the same 10000 samples in the similar k-fold cross-validation but this time your parameters will be fixed. You can choose to run k-fold cross-validation once and get k error measures for each of the subset; 2*k if you consider training set which you could also look into.

How is feature selection used in regression modeling?

Feature selection is the process of identifying and selecting a subset of input variables that are most relevant to the target variable. Perhaps the simplest case of feature selection is the case where there are numerical input variables and a numerical target for regression predictive modeling.

When do you need to use cross validation?

The cross-validation is only used to estimate the predictive performance of the single model trained on the whole dataset. It is VITAL in using cross-validation that in each fold you repeat the entire procedure used to fit the primary model, as otherwise you can end up with a substantial optimistic bias in performance.