Contents
In general, it is recommended to avoid having correlated features in your dataset. Indeed, a group of highly correlated features will not bring additional information (or just very few), but will increase the complexity of the algorithm, thus increasing the risk of errors.
Is correlation important in classification?
Given the statistical nature of most methods in machine learning, it, therefore, makes sense to speak, in general, of correlation but not of causality. The phenomena of correlation don’t influence the structure and way of working of a classifier.
Why is multicollinearity not good?
However, severe multicollinearity is a problem because it can increase the variance of the coefficient estimates and make the estimates very sensitive to minor changes in the model. The result is that the coefficient estimates are unstable and difficult to interpret.
How are correlated features affect the classification accuracy?
I’d like to ask everyone a question about how correlated features (variables) affect the classification accuracy of machine learning algorithms. With correlated features I mean a correlation between them and not with the target class (i.e the perimeter and the area of a geometric figure or the level of education and the average income).
Which is better, highly correlated features or uncorrelated features?
In general, I’d say the more uncorrelated the features are, the better the classifier performance is going to be. Given a set of highly correlated features, it may be possible to use PCA techniques to make them as orthogonal as possible to improve classifier performance.
When we have highly correlated features in the dataset, the values in “S” matrix will be small. So inverse square of “S” matrix (S^-2 in the above equation) will be large which makes the variance of Wₗₛ large. So, it is advised that we keep only one feature in the dataset if two features are highly correlated.
There are three main reasons why you would remove correlated features: Due to the curse of dimensionality, less features usually mean high improvement in terms of speed. The keyword being harmful. If you have correlated features but they are also correlated to the target, you want to keep them.