Contents
You could also define correlation is a measure of how strongly one variable depends on another. A high correlation is often a useful property—if two variables are highly correlated, we can predict one from the other.
Why is correlation useful?
Not only can we measure this relationship but we can also use one variable to predict the other. For example, if we know how much we’re planning to increase our spend on advertising then we can use correlation to accurately predict what the increase in visitors to the website is likely to be.
To summarise, Xgboost does not randomly use the correlated features in each tree, which random forest model suffers from such a situation.
How should I deal with highly correlated features?
What are the implications of this high correlations, and how do I perform my feature engineering knowing this?
Why do we exclude highly correlated features when building?
When we have highly correlated features in the dataset, the values in “S” matrix will be small. So inverse square of “S” matrix (S^-2 in the above equation) will be large which makes the variance of Wₗₛ large. So, it is advised that we keep only one feature in the dataset if two features are highly correlated.
How are features chosen with the highest correlation?
Therefore, the feature pair with the highest absolute correlation coefficient is selected. The feature of this pair which has the lower correlation with the passengers’ survival is eliminated. This procedure is repeated until no features are left. The revesre order of elimination shall be the order of selecting features.
How to handle correlation features in Kaggle kernel?
The produced data set contains a lot of strongly correlated features. In this kernel I analyse these correlation. Furthermore, I implement different strategies to reduce the number of features and test these strategies with respect to the achievable prediction accuracy of different classifiers.