How do you compare models with cross validation?
A general way to perform model comparison is cross-validation [Hastie2008]. In this method, a model is fit to some of the data (a learning set) and the model is then used to predict a held-out set (a testing set). The model predictions can then be compared to estimate prediction error on the held out set.
What comes first verification or validation?
Verification checks whether the software confirms a specification whereas Validation checks whether the software meets the requirements and expectations. Comparing Verification vs Validation testing, Verification process comes before validation whereas Validation process comes after verification.
Why do you use cross validation and test?
The cross validation result is therefore likely to be too optimistic. For the same reason, you cannot drop the cross validation set and use the test set for selecting hyper parameters, because then you are pretty much guaranteed to be overestimating how good your model is.
Why do you use both validation set and test set?
One approach that is reasonable is to re-use (train + cv) data to re-train using the hyper-params you have found, before testing. That way you do get to train on more data, and you still get an independent measure of performance at the end. If you want to get more out of cross validation, the usual approach is k-fold cross validation.
Can you run k-fold cross validation more than once?
Once model is selected again use the same 10000 samples in the similar k-fold cross-validation but this time your parameters will be fixed. You can choose to run k-fold cross-validation once and get k error measures for each of the subset; 2*k if you consider training set which you could also look into.
When to divide data into training and validation?
If the test set is locked away, but you still want to measure performance on unseen data as a way of selecting a good hypothesis, then divide the available data (without the test set) into a training set and a validation set.