Class Imbalance
When one class vastly outnumbers another — and the standard advice to resample is mostly wrong.
When not to use it
- (Resampling, that is.)*
- Before adjusting your threshold. That's free, preserves calibration, and usually suffices.
- When you need calibrated probabilities. Resampling destroys them by construction.
- On the validation or test set. Ever. It produces beautiful meaningless scores.
- SMOTE in high dimensions. Interpolating between rare points synthesises examples in regions where nothing real lives.
Reach for something else instead
- Threshold adjustment — the correct move. Free, and it's just doing the decision theory.
- Cost-weighted loss — change the objective, not the data.
- PR curves instead of accuracy — most imbalance problems are metric problems.
- Anomaly detection framing — if the minority is truly rare, it may be the wrong model class.
Sources & further reading
- Chawla et al. (2002), SMOTE: Synthetic Minority Over-sampling Technique — the method; read it, then read what came after.
- Van den Goorbergh et al. (2022), The harm of class imbalance corrections for risk prediction models — imbalance correction damages calibration and doesn't improve discrimination.
- He & Garcia (2009), Learning from Imbalanced Data — the survey that frames it properly as a decision problem.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reaching for SMOTE first. Threshold adjustment is free, correct, and usually enough.
- Resampling before the train/test split. Classic, and the scores are fiction.
- Not noticing calibration is gone after resampling.
- Blaming imbalance for what's a base-rate problem. At 0.1% prevalence, most positives are false regardless of your model.
At a glance
FieldMachine Learning
The real problema decision rule, not the learning
The fix, in ordermetric, threshold, loss weights, then maybe resample
Why resampling is poorit destroys calibration and interpolates into empty space
When it genuinely helpsextreme imbalance where batches contain no minority examples
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Resampling vs. threshold adjustment — one fabricates data and breaks your probabilities; the other does the decision theory you skipped, for free.