ROC and AUC
A curve showing every threshold at once, summarised into one number — the most-reported classification metric, and it has a coherence problem almost nobody knows about.
When not to use it
- On imbalanced data. The FPR denominator is huge, so false positives barely register. Use a PR curve.
- To compare two models, strictly. Hand's result: the implicit cost weighting differs per model, so the comparison isn't on a common scale.
- When you need probabilities. AUC is rank-based. A perfectly-ranking, wildly-miscalibrated model scores 1.0.
- As a substitute for choosing a threshold. You have to ship one, and AUC won't tell you which.
Reach for something else instead
- PR-AUC — the honest curve on imbalanced problems.
- H-measure — Hand's coherent alternative; fixes the cost distribution explicitly. Better, unused.
- Partial AUC — integrate only the region you'd operate in, rather than thresholds you'd never use.
- Cost-weighted error at your actual threshold — the number that corresponds to a decision.
The classifier never changes. Only the class mix does.
AUC is rank-based: it asks whether positives score above negatives, which is a question the class mix cannot affect. Precision asks how many of your alarms were real, which the mix affects enormously. That is why AUC survives imbalance looking excellent while the model becomes useless in production. Every number here is computed from 4,000 seeded samples and a threshold fixed at 0.50.
The full account
Where it came from
ROC analysis is not a machine-learning invention and predates the field by decades. The curve was developed by radar operators during the Second World War, working on a problem with the same shape as any classifier's: a blip on a screen is either an aircraft or noise, and the operator must set a threshold for calling it. Set it low and you scramble fighters at flocks of birds; set it high and you miss the bombers. The "receiver operating characteristic" was literally the operating characteristic of a radio receiver — a curve describing how its hit rate and false-alarm rate traded off as you turned the sensitivity dial.
Green and Swets formalised this as signal detection theory in 1966, and the framework's central insight is one that machine learning has partly forgotten: it separates the discriminability of the signal from the decision criterion the operator chooses. Those are different things. Discriminability is a property of the evidence; the criterion is a policy choice about which error you would rather make. ROC's whole purpose was to hold the first constant while varying the second.
Hanley and McNeil brought it to medicine in 1982 and supplied the interpretation everyone quotes: the area under the curve equals the probability that a randomly chosen positive case receives a higher score than a randomly chosen negative one. That is an elegant result, and it is also where the trouble starts, because a statement about randomly chosen pairs is not a statement about your data.
What AUC actually measures
The probabilistic reading is precise and worth restating carefully. AUC is the probability that a random positive outranks a random negative — where random means drawn uniformly from each class. It says nothing about how many of each class exist.
That last clause is the entire problem, and it is not a subtlety. Because AUC samples one positive against one negative, it is computed on a balanced 1:1 comparison no matter what your actual base rate is. Move from 50% positives to 1% positives and the ROC curve barely shifts, because both of its axes are normalised within class: the true positive rate divides by the positives, the false positive rate divides by the negatives. Each class is graded on its own curve. The ratio between them — the thing that determines whether your model is useful — never enters the calculation.
Precision does not have that property. Precision asks: of the cases I flagged, how many were real? That question depends on how many negatives there are to falsely flag. At a 1% base rate, a false positive rate that looks negligible against 99,000 negatives produces a flood of false alarms that swamps the 1,000 true positives. The figure above computes this live: hold discriminability fixed, drag the base rate, and watch AUC sit still while precision falls off a cliff. Neither number is lying. They are answering different questions, and only one of them is the question you asked.
The critique, which is older and sharper than most people know
The case against AUC is not a blog-post opinion. It has been made independently, in three separate fields, by people who had no particular reason to agree.
David Hand's 2009 argument is the most fundamental and the least understood. His objection is not that AUC is imprecise — it is that AUC is incoherent. Averaging performance across all possible thresholds implicitly assigns weights to the relative cost of false positives and false negatives. Those weights depend on the shape of the score distribution, which differs between classifiers. So comparing two models by AUC compares them using different cost assumptions for each model — as Hand puts it, it is like judging two athletes using a scoring system that changes depending on which athlete you are looking at. Given that a real deployment has one cost ratio, and you usually know roughly what it is, evaluating at all thresholds equally is not a neutral choice. It is a specific and usually wrong one.
Davis and Goadrich supplied the formal relationship in 2006. Their dominance theorem is genuinely useful and routinely misquoted: a curve that dominates in ROC space also dominates in PR space, and vice versa. What people take from this is "the two are equivalent, so use either." What it actually says is that dominance transfers — and dominance is rare. Real classifiers cross. When curves cross, the ranking depends entirely on which space you plot in, and optimising area in one space does not optimise area in the other.
Saito and Rehmsmeier made the empirical case in 2015 and it is the most directly practical of the three: on imbalanced data, the ROC plot is visually misleading. It looks encouraging while the classifier is useless, because the false positive rate axis is compressed by the enormous negative class. The PR plot shows the same classifier failing.
The convergence is the striking part. Lobo and colleagues reached substantially the same conclusion in 2008 from ecological modelling, in a paper titled with no ambiguity at all: AUC: A Misleading Measure. Signal detection, medical statistics, machine learning and ecology each adopted AUC, each used it for years, and each independently produced a paper arguing it was being misapplied in exactly the same way.
| What it asks | Sensitive to base rate? | Fails when | |
|---|---|---|---|
| AUC | Does a random positive outrank a random negative? | No — each axis is normalised within class | The classes are imbalanced, or you have one operating threshold |
| Average precision (PR-AUC) | Of what I flagged, how much was real, across thresholds? | Yes | You genuinely care about both classes equally |
| Precision & recall at your threshold | What actually happens when I ship this? | Yes | You have not chosen a threshold yet |
| Expected cost | What does this cost me? | Yes | You cannot estimate the cost ratio |
Why it survives anyway
If the critique is this well-established, the obvious question is why AUC remains the default in nearly every paper and leaderboard. The answer is not that researchers are unaware; it is that AUC has three properties that are genuinely convenient and one that is genuinely valuable.
It is threshold-free, so you can report a model before deciding how to deploy it. It is a single number, so it sorts a leaderboard. It is invariant to class balance, which — and this is the part worth keeping — makes it comparable across datasets with different base rates. That invariance is a real feature when you are asking a research question about discriminability. It becomes a defect the moment you are asking a deployment question about utility, because it discards precisely the information that determines whether the model is worth running.
So the honest summary is that AUC answers a research question well and a product question badly, and the field's mistake is not using it. The mistake is reporting it as though it settled something it does not address.
What to do instead
The practical advice follows from the diagnosis and is unglamorous. If you have a deployment threshold, report precision and recall at that threshold, because that is what will happen. If you do not have one yet but your data is imbalanced, report average precision, which is the PR-space analogue and does move with the base rate. If you know your cost ratio — and you usually know it better than you admit — evaluate expected cost directly, which is what the ROC framework was invented to let you do in the first place.
And if you report AUC, report the base rate beside it. The number is not wrong. It is incomplete in a specific way, and a reader who knows the base rate can supply what is missing. A reader who does not know it cannot tell the difference between a model that works and the figure above at 1%.
Further reading
- Fawcett (2006), An Introduction to ROC Analysis — the standard reference, and it's genuinely clear.
- Hand (2009), Measuring Classifier Performance: A Coherent Alternative to the Area Under the ROC Curve — AUC uses different cost weightings for different classifiers. The critique that should be famous.
- Saito & Rehmsmeier (2015), The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets — the imbalance problem, demonstrated.
- Davis & Goadrich (2006), The Relationship Between Precision-Recall and ROC Curves — proves the dominance theorem: a curve dominating in ROC space dominates in PR space, and vice versa.
- Hanley & McNeil (1982), The Meaning and Use of the Area Under a Receiver Operating Characteristic (ROC) Curve — the paper that established AUC in medicine, and the source of its probabilistic interpretation.
- Lobo, Jiménez-Valverde & Real (2008), AUC: A Misleading Measure of the Performance of Predictive Distribution Models — the ecology field's independent arrival at the same critique.
- Green & Swets (1966), Signal Detection Theory and Psychophysics — the framework ROC analysis came from, two decades before machine learning existed.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reporting ROC-AUC on a heavily imbalanced problem. It flatters, and this is the most common misuse.
- Comparing two AUCs as if they're on the same scale. Hand showed they aren't.
- Reading high AUC as "well-calibrated." It's a ranking metric; calibration is invisible to it.
- Integrating over thresholds you'd never use. Partial AUC exists for exactly this.
At a glance
Often compared with
Where this sits
A destination. 3 concepts lead here, and nothing in the corpus depends on it.
Computed from the prerequisite graph, not assigned. How this works