Class Imbalance
When one class vastly outnumbers another — and the standard advice to resample is mostly wrong.
When not to use it
- (Resampling, that is.)*
- Before adjusting your threshold. That's free, preserves calibration, and usually suffices.
- When you need calibrated probabilities. Resampling destroys them by construction.
- On the validation or test set. Ever. It produces beautiful meaningless scores.
- SMOTE in high dimensions. Interpolating between rare points synthesises examples in regions where nothing real lives.
Reach for something else instead
- Threshold adjustment — the correct move. Free, and it's just doing the decision theory.
- Cost-weighted loss — change the objective, not the data.
- PR curves instead of accuracy — most imbalance problems are metric problems.
- Anomaly detection framing — if the minority is truly rare, it may be the wrong model class.
The real problem is a decision rule, not the learning.
One classifier, trained once, never retrained — 2,000 seeded scores at 2% prevalence. At the default threshold it calls everything negative and scores 98% accuracy, which is exactly what you get by predicting “no” forever. Nothing was learned badly; the model's scores are fine, and the ranking they produce is good. What is broken is the line drawn through them, and moving that line costs nothing. Drag the threshold down and recall appears out of the model you already had. This is why threshold adjustment comes before SMOTE: it is free, it is reversible, and it leaves calibration intact.
The full account
The advice everyone gives
Your data is 1% positives. Every tutorial, every course, every accepted Stack Overflow answer says the same thing: the classes are imbalanced, so balance them. Oversample the minority, undersample the majority, or — best of all, because it is cleverer — use SMOTE, which synthesises new minority examples by interpolating between existing ones rather than duplicating them.
SMOTE has been cited many thousands of times since 2002. It is in every library. It is the default recommendation on essentially every page you will find about imbalanced data. And the best evidence available says that for a large class of real problems it makes your model worse, in a way you will not see unless you look at the one metric nobody checks.
What the simulation found
Van den Goorbergh, van Smeden, Timmerman and Van Calster ran the test properly in 2022 — Monte Carlo simulation across event fractions, training set sizes and predictor counts, plus a real case study on ovarian cancer diagnosis. They compared no correction against random undersampling, random oversampling, and SMOTE, and measured three things rather than one: discrimination, calibration, and classification.
Discrimination did not improve. AUC after correction was no better than without it, and for random undersampling it was often worse — with the gap widening exactly where you'd reach for correction hardest: lower event fraction, smaller training set, more predictors.
Calibration was destroyed. This is the finding, and the numbers are not subtle. Models trained on uncorrected data had median calibration intercepts between −0.05 and 0.03 — essentially perfect. After correction, at a 1% event fraction, median calibration intercepts were −4.5 or lower. At 10%, −2.1 or lower. At 30%, −0.7 or lower. The model now systematically and enormously overestimates the probability that a case belongs to the minority class, because you told it the minority class was common and it believed you.
And the classification benefit was free anyway. Correction did improve the sensitivity/specificity balance — that part of the folklore is true. But the authors got the same improvement by leaving the data alone and shifting the probability threshold, which costs nothing, breaks nothing, and is reversible.
Their conclusion is the sentence to carry away: outcome imbalance is not a problem in itself, and imbalance correction may even worsen model performance.
| Discrimination (AUC) | Calibration intercept @ 1% events | Sensitivity/specificity balance | |
|---|---|---|---|
| No correction | baseline | −0.05 to 0.03 | unbalanced at threshold 0.5 |
| Random oversampling | no better | ≤ −4.5 | improved |
| Random undersampling | often worse | ≤ −4.5 | improved |
| SMOTE | no better | ≤ −4.5 | improved |
| Shift the threshold | unchanged | unchanged | improved — same benefit, no damage |
Why this went unnoticed for twenty years
Because almost nobody measures calibration. The field reports AUC, and AUC is invariant to class balance by construction — each of its axes is normalised within class, so resampling barely moves it. Then it reports accuracy, precision, recall, F1: all of them are computed at a threshold, and all of them respond to the sensitivity/specificity shift that correction genuinely produces. Every metric on the standard dashboard either can't see the damage or shows the one real improvement.
Calibration is the metric that sees it, and calibration is the one people skip — because it needs a reliability diagram rather than a number, and because for years the field treated probability estimates as a means to a ranking rather than as output in their own right.
That is fine right up until someone reads the probability. A clinician deciding whether a 4% risk warrants surgery is reading the probability. A credit model pricing a loan is reading the probability. A fraud team allocating investigators by expected loss is reading the probability. Give those people a model with a calibration intercept of −4.5 and every number they see is wrong in the same direction, and their decisions are wrong with it — while the AUC in your report looks excellent, because it is.
A real-world replication in 2026 found the effect intact outside simulation, with risk over-prediction reaching roughly 63% across a set of clinical tasks. This is not a theoretical concern that a better resampler will fix.
What the figure shows, and what it doesn't
The figure on this page demonstrates the other half of the story: at a 1% base rate, a model with 98% accuracy and 10% recall is what the default threshold gives you, and the accuracy number is meaningless because predicting "no" always scores 99%. That part of the folklore is correct — imbalanced data does wreck naive metrics.
The mistake is what follows from it. The correct inference is stop using accuracy and move the threshold. The inference the field made is change the data, and the data was not the problem. The base rate is a fact about the world; a model that knows the base rate is a model that knows something true. Resampling teaches it something false, in exchange for a threshold shift you could have done for free.
What to do instead
Leave the data alone. Move the threshold to wherever your cost ratio says it belongs — false negatives and false positives have different prices, you know roughly what they are, and the threshold is the parameter that encodes them.
Use class weights rather than resampling if you want the loss to reflect the cost, since it does not fabricate or discard rows.
Measure calibration. A reliability diagram takes one line and it is the only chart that would have caught this.
And if you have already deployed a SMOTE-trained model, the authors tested the repair: recalibrating on the original imbalanced data restored median calibration intercepts to between −0.07 and 0.03. The damage is undoable — but only if you look for it first.
Further reading
- Chawla et al. (2002), SMOTE: Synthetic Minority Over-sampling Technique — the method; read it, then read what came after.
- Van den Goorbergh et al. (2022), The harm of class imbalance corrections for risk prediction models — imbalance correction damages calibration and doesn't improve discrimination.
- He & Garcia (2009), Learning from Imbalanced Data — the survey that frames it properly as a decision problem.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reaching for SMOTE first. Threshold adjustment is free, correct, and usually enough.
- Resampling before the train/test split. Classic, and the scores are fiction.
- Not noticing calibration is gone after resampling.
- Blaming imbalance for what's a base-rate problem. At 0.1% prevalence, most positives are false regardless of your model.
At a glance
Often compared with
Where this sits
3 concepts come first. Understanding it opens up 1 more.
Computed from the prerequisite graph, not assigned. How this works