How do you deal with highly unbalanced data?
7 Techniques to Handle Imbalanced Data
- Use the right evaluation metrics.
- Resample the training set.
- Use K-fold Cross-Validation in the right way.
- Ensemble different resampled datasets.
- Resample with different ratios.
- Cluster the abundant class.
- Design your own models.
Is it possible to cross validation an imbalanced dataset?
Namely, the minority class is simply a rare event, which makes finding data that would hopefully balance the class distribution of the dataset very difficult. Because of these disadvantages, there can be caveats when trying to leverage cross-validation with an imbalanced dataset. These caveats are perhaps best illustrated with an example.
When to do cross validation when upsampling data?
We have about 78% recall on one of our models before we have tried oversampling. This is the number to beat. Normally we would wait until we had finished our modeling to look at the test set, but an important part of this is to see how oversampling, done incorrectly, can make us too confident in our ability to generalize based off cross-validation.
How to cross validation for highly imbalanced train data?
Fit model on undersampled train data and calcualte the metric of interest on the test set. Consider the other Fold as test set (this time e.g., K=2) and the rest as train set (K=1,3,4,5). Undersample train set and proceed to step 3. Continue this procedure for the rest of the folds.
How is random undersampling used in imbalanced learning?
Random undersampling involves randomly selecting examples from the majority class and deleting them from the training dataset. In the random under-sampling, the majority class instances are discarded at random until a more balanced distribution is reached. — Page 45, Imbalanced Learning: Foundations, Algorithms, and Applications, 2013