ROC and AUC
A curve showing every threshold at once, summarised into one number — the most-reported classification metric, and it has a coherence problem almost nobody knows about.
When not to use it
- On imbalanced data. The FPR denominator is huge, so false positives barely register. Use a PR curve.
- To compare two models, strictly. Hand's result: the implicit cost weighting differs per model, so the comparison isn't on a common scale.
- When you need probabilities. AUC is rank-based. A perfectly-ranking, wildly-miscalibrated model scores 1.0.
- As a substitute for choosing a threshold. You have to ship one, and AUC won't tell you which.
Reach for something else instead
- PR-AUC — the honest curve on imbalanced problems.
- H-measure — Hand's coherent alternative; fixes the cost distribution explicitly. Better, unused.
- Partial AUC — integrate only the region you'd operate in, rather than thresholds you'd never use.
- Cost-weighted error at your actual threshold — the number that corresponds to a decision.
Sources & further reading
- Fawcett (2006), An Introduction to ROC Analysis — the standard reference, and it's genuinely clear.
- Hand (2009), Measuring Classifier Performance: A Coherent Alternative to the Area Under the ROC Curve — AUC uses different cost weightings for different classifiers. The critique that should be famous.
- Saito & Rehmsmeier (2015), The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets — the imbalance problem, demonstrated.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reporting ROC-AUC on a heavily imbalanced problem. It flatters, and this is the most common misuse.
- Comparing two AUCs as if they're on the same scale. Hand showed they aren't.
- Reading high AUC as "well-calibrated." It's a ranking metric; calibration is invisible to it.
- Integrating over thresholds you'd never use. Partial AUC exists for exactly this.
At a glance
FieldMachine Learning
ROCtrue positive rate vs. false positive rate, across all thresholds
AUC's nice interpretationP(random positive scores above random negative)
Flatters onimbalanced data
Hand's critiquethe implicit cost weighting differs per classifier, so cross-model comparison is incoherent
Blind tocalibration
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
ROC vs. PR curves — one divides false positives by all the negatives and flatters; the other divides by predicted positives and doesn't. On imbalanced data only the second is honest.