Should feature selection be done before train test split or after?

Should feature selection be done before train test split or after?

The conventional answer is to do it after splitting as there can be information leakage, if done before, from the Test-Set. Moreover, feature importance scores can only be evaluated when, given a set of instances rather than a single test/unknown instance.

Should correlated features be removed?

In general, it is recommended to avoid having correlated features in your dataset. Indeed, a group of highly correlated features will not bring additional information (or just very few), but will increase the complexity of the algorithm, thus increasing the risk of errors.

Should you scale before or after splitting data?

Yes, scaling should be done on both the training data and the test data. Additionally, the scaling should be the same. If you scale the training set one way and the testing set another way, this will still create issues.

How do you remove features with high correlation?

To remove the correlated features, we can make use of the corr() method of the pandas dataframe. The corr() method returns a correlation matrix containing correlation between all the columns of the dataframe.

Is correlation between features good or bad?

So, why is correlation useful? Correlation can help in predicting one attribute from another (Great way to impute missing values). Correlation can (sometimes) indicate the presence of a causal relationship.

Should I scale the test set?

The test set must use identical scaling to the training set. And the point is given that: Do not scale the training and test sets using different scalars: this could lead to random skew in the data.

How do you reduce correlated features?

The easiest way is to delete or eliminate one of the perfectly correlated features. Another way is to use a dimension reduction algorithm such as Principle Component Analysis (PCA).

What is the P value in correlation?

A p-value is the probability that the null hypothesis is true. In our case, it represents the probability that the correlation between x and y in the sample data occurred by chance. A p-value of 0.05 means that there is only 5% chance that results from your sample occurred due to chance.

How to split data into training and test set?

You first need to split the data into training and test set (validation set could be useful too). Don’t forget that testing data points represent real-world data.

When to do normalization after a test split?

As @Erwan said, you should normalize the training set and then use the same normalization steps on the test set. So your code should look like: Answer to your question: Do Normalization after splitting into train and test/validation. The reason is to avoid any data leakage.

Why are test sets not used in training?

This is because the test set plays the role of fresh unseen data, so it’s not supposed to be accessible at the training stage. Using any information coming from the test set before or during training is a potential bias in the evaluation of the performance.

What is the division between training and test set?

The division between training and test set is an attempt to replicate the situation where you have past information and are building a model which you will test on future as-yet unknown information: the training set takes the place of the past and the test set takes the place of the future, so you only get to test your trained model once.