Contents
What is nested resampling?
Nested resampling uses an additional layer of resampling that separates the tuning activities from the process used to estimate the efficacy of the model. Once the tuning results are complete, a model is fit to each of the outer resampling splits using the best parameter associated with that resample.
What is resampling in machine learning?
Data is the currency of applied machine learning. Resampling is a methodology of economically using a data sample to improve the accuracy and quantify the uncertainty of a population parameter. Resampling methods, in fact, make use of a nested resampling method.
How is nested resampling used in a model?
Nested resampling does an additional layer of resampling that separates the tuning activities from the process used to estimate the efficacy of the model. An outer resampling scheme is used and, for every split in the outer resample, another full set of resampling splits are created on the original analysis set.
When to use an outer or inner resampling scheme?
An outer resampling scheme is used and, for every split in the outer resample, another full set of resampling splits are created on the original analysis set. For example, if 10-fold cross-validation is used on the outside and 5-fold cross-validation on the inside, a total of 500 models will be fit.
How to calculate the error rate of a nested resampling?
The wrapped Learner ( makeLearner ()) is generated by calling makeTuneWrapper (). Note that in practice the parameter set should be larger. A common recommendation is 2^ (-12:12) for both C and sigma. You can obtain the error rates on the 3 outer test sets by:
How to get nested resampling for free in MLR?
In mlr, you can get nested resampling for free without programming any looping by using the wrapper functionality. This works as follows: Generate a wrapped Learner (makeLearner()) via function makeTuneWrapper() or makeFeatSelWrapper(). Specify the inner resampling strategy using their resampling argument.