Contents
The potential solutions include the following:
- Remove some of the highly correlated independent variables.
- Linearly combine the independent variables, such as adding them together.
- Perform an analysis designed for highly correlated variables, such as principal components analysis or partial least squares regression.
Correlation means association – more precisely it is a measure of the extent to which two variables are related. Therefore, when one variable increases as the other variable increases, or one variable decreases while the other decreases. An example of positive correlation would be height and weight.
What’s the difference between correlation and linear regression?
Before going into complex model building, looking at data relation is a sensible step to understand how your different variable interact together. Correlation look at trends shared between two variables, and regression look at relation between a predictor (independent variable) and a response (dependent) variable.
When is there no correlation between two variables?
The other way round when a variable increase and the other decrease then these two variables are negatively correlated. In the case of no correlation no pattern will be seen between the two variable.
The correlation also has nothing to do with heteroskedasticity. The key is that the response variable = the estimated regression function + the residual, so it makes sense that the response variable would be positively correlated with the residual, since it’s made up in part by it.
What are the residuals of the regression line?
Each observation will have a residual. If an observation is above the regression line, then its residual, the vertical distance from the observation to the line, is positive. Observations below the line have negative residuals. One goal in picking the right linear model is for these residuals to be as small as possible.