Contents
Why do we get strong correlations in significance tests?
Simply using significance tests and a stepwise procedure to perform model selection can lead you to believe that you have a very strong model with significant predictors when you, in fact, do not; you may get strong correlations by chance and these correlations can seemingly be enhanced as you remove other unnecessary predictors.
Using significance testing we are not directly concerned with the “performance” of the model under the null hypothesis versus other models, but we are concerned with documenting that the null is wrong.
Which is an example of a cross validation algorithm?
It depends, for instance, on the choice of a test statistic. Cross validation is really an algorithm for estimation of the expected generalization error, which is the important general concept, and which depends on the choice of a loss function.
Is the AIC asympotically equivalent to cross validation?
Other respondents might mention that stepwise procedures using the AIC or BIC are asympotically equivalent to cross validation. This only works as the number of observations relative to the number of predictors gets large, however.
What is the purpose of cross validation in machine learning?
The Purpose of Cross Validation The purpose of cross validation is to assess how your prediction model performs with an unknown dataset. We shall look at it from a layman’s point of view. You are learning how to drive a car.
How are statistical significance tests used in machine learning?
Comparing machine learning models via statistical significance tests imposes some expectations that in turn will impact the types of statistical tests that can be used; for example: Skill Estimate. A specific measure of model skill must be chosen.
How is significance testing carried out in different contexts?
To be crystal clear about the terminology, significance testing is a general concept, which is carried out differently in different contexts. It depends, for instance, on the choice of a test statistic.