ROC AUC looks great on imbalanced data and tells you almost nothing

Rare-event problems are most of what I have worked on. Equipment failures, safety violations, fraud. The positive class is a fraction of a percent, and a model with a beautiful 0.95 ROC AUC turns out to be useless in production.

The reason is in the denominators.

False positive rate is FP / (FP + TN). On imbalanced data, TN is enormous, so even a large pile of false positives barely moves the number. The ROC curve stays flattered by the majority class.

Precision is TP / (TP + FP). The majority class appears nowhere. Every false positive costs you directly. That is why a precision-recall curve stays honest when a ROC curve does not, which Saito and Rehmsmeier show clearly.

Concretely: one in a thousand samples is positive, and at your operating point you catch 90 percent of them while flagging 2 percent of the negatives. The FPR is 0.02, which looks excellent on a ROC curve. Run it over a thousand samples: one positive, and you catch 0.9 of it, against 999 negatives of which 2 percent gives you about 20 false alarms. Precision is 0.9 / 20.9, a little over 4 percent. An inspector following your alerts is wrong roughly 23 times out of every 24. Same model, same numbers, completely different story.

What I do instead:

• Report PR AUC alongside ROC AUC, and lead with PR when prevalence is low.
• Quote the baseline. A random classifier's PR AUC is the positive rate, so 0.05 PR AUC at 1 percent prevalence is real signal and 0.05 at 20 percent prevalence is nothing. ROC AUC has a fixed 0.5 baseline, which is exactly why it feels reassuring and is not.
• Pick the operating point with whoever handles the alerts, in units of their time. "How many false alarms per shift is tolerable" gets a better threshold than any curve does.