Why is ROC bad for Imbalanced data?
Although widely used, the ROC AUC is not without problems. For imbalanced classification with a severe skew and few examples of the minority class, the ROC AUC can be misleading. This is because a small number of correct or incorrect predictions can result in a large change in the ROC Curve or ROC AUC score.
Why is precision recall curve better for Imbalanced data?
FPR is considered better when it’s smaller since it indicates fewer false positives. In imbalanced data, the FPR tends to stay at small values due to the large numbers of negatives (i.e. making the denominator large). Thus, FPR becomes less informative for the model performance in this situation.
When is a binary classification problem highly imbalanced?
This article assumes that the readers have some knowledge about binary classification problems. Consider a binary classification problem where the target variable is highly imbalanced.
Can a high accuracy model be used for imbalanced classification?
Although widely used, classification accuracy is almost universally inappropriate for imbalanced classification. The reason is, a high accuracy (or low error) is achievable by a no skill model that only predicts the majority class. For more on the failure of classification accuracy, see the tutorial:
Which is better for imbalanced data ROC or PR?
The classification_report function is really useful here in this case: Precision-Recall (PR) curve is recommended over ROC for imbalanced data. As the figure shown below, ROC curves can overestimate the performance of classifier on imbalanced data whereas PR curves reveal the drop in precision at the same level of recall for the imbalanced data.
How to know if a binary classifier is accurate?
There are many metrics for evaluating how good a binary classifier is doing in predicting the class labels for instances/examples. Below are some caveats and suggestions for choosing and interpreting the appropriate metrics. Accuracy can be misleading.