Contents
How are time series used in cross validation?
Time series datasets associated to such problems have two timestamps attached to each sample: a prediction time, when the machine learning model has to make a prediction, and an evaluation time, when the response becomes available and the prediction error can be computed. We will discuss cross-validation strategies adapted to this kind of problems.
Which is a feature of combinatorial cross validation?
A nice feature of combinatorial cross-validation is also that as each block of samples appears the same number of times in the validation set, we can group them (arbitrarily) into validation predictions over the full dataset (keeping in mind that these predictions have been made by models trained on different train sets).
How are time series data used in machine learning?
Developing machine learning models for time series data requires special care, mainly because the usual machine learning assumption that the samples are independent generally does not hold.
Why are time series data often strongly correlated?
The time series data is often strongly correlated along the time axis (think about the GoogleMap example: a traffic jam affects all the users on the same route at a given time). The randomization will make it likely that for each sample in the validation set, numerous strongly correlated samples exist in the train set.
When to use normal cross validation in prediction?
If the submodels are stable, there’s no difference in the prediction (just a waste of computational resources). Normal cross validation compares un-aggregated predictions to the ground truth, so it doesn’t evaluate possible stabilization by aggregating.
When to leave one data point out of cross validation?
Leave One Out Cross Validation (LOOCV): This approach leaves 1 data point out of training data, i.e. if there are n data points in the original sample then, n-1 samples are used to train the model and p points are used as the validation set.
Which is the best strategy for cross validation?
Cross-validation is a well-established methodology for choosing the best model by tuning hyper-parameters or performing feature selection. There are a plethora of strategies for implementing optimal cross-validation.