Contents
What correlation is significant?
In most research the threshold to what we consider statistically significant is a p-value of 0.05 or below and it’s called the significance level α. So we can set our significance level to 0.05 (α =0.05) and find the P-value.
How do you explain a correlation matrix?
A correlation matrix is a table showing correlation coefficients between variables. Each cell in the table shows the correlation between two variables. A correlation matrix is used to summarize data, as an input into a more advanced analysis, and as a diagnostic for advanced analyses.
Is it useful to have highly correlated features?
Just because your features are correlated does not mean they are not useful, in fact, this correlation could be valuable if your dataset is in fact representative of what is out there “in the wild”.
When we have highly correlated features in the dataset, the values in “S” matrix will be small. So inverse square of “S” matrix (S^-2 in the above equation) will be large which makes the variance of Wₗₛ large. So, it is advised that we keep only one feature in the dataset if two features are highly correlated.
In theory, this should not affect your ability to make predictions – after all, the only truly useless data would be a restated column (or a column whose values can be directly derived from some other – e.g. having radius and circumference in two columns).
Is the variance of highly correlated features large?
Now you should have guessed that when we have highly correlated features, the variance of Wₗₛ will be large. Yes, the guess is right !! But let us see how this is correct mathematically. Any n x d matrix can be decomposed as The above decomposition is called “Singular Value Decomposition”.