Contents
When to do feature selection before or after split?
The contradicting answer is that, if only the Training Set chosen from the whole dataset is used for Feature Selection, then the feature selection or feature importance score orders is likely to be dynamically changed with change in random_state of the Train_Test_Split.
Which is the best method for feature selection?
F-Test captures the linear relationship well. Mutual Information captures any kind of relationship between two variables. http://scikit-learn.org/stable/auto_examples/feature_selection/plot_f_test_vs_mi.html This method removes features with variation below a certain cutoff.
Is it possible to generalize the feature selection for any particular work?
And if the feature selection for any particular work changes, then no Generalization of Feature Importance can be done, which is not desirable.
Are there any drawbacks to feature selection?
But the main drawbacks of wrapper methods is the sheer amount of models that needs to be trained. It is computationally very expensive and is infeasible with large number of features. Feature selection can also be acheived by the insights provided by some Machine Learning models.
What is the problem of data leakage in predictive modeling?
Data leakage is when information from outside the training dataset is used to create the model. In this post you will discover the problem of data leakage in predictive modeling. What is data leakage is in predictive modeling. Signs of data leakage and why it is a problem.
How can you tell if you have data leakage?
An easy way to know you have data leakage is if you are achieving performance that seems a little too good to be true. Like you can predict lottery numbers or pick stocks with high accuracy. Data leakage is generally more of a problem with complex datasets, for example:
When to check for data leakage in machine learning?
Once you have completed your modeling process and actually created your final model, evaluate it on the validation dataset. This can give you a sanity check to see if your estimation of performance has been overly optimistic and has leaked.