Train/Test Split
Holding back data the model never sees, so you can find out whether it learned anything or just memorised.
When not to use it
- Random splitting on time-series data. You'd be training on the future. Split by date, without exception.
- Random splitting when rows are grouped. Multiple records per customer across the boundary means the model learns the customer, not the pattern.
- As your only evidence. A clean split measures held-out performance, not deployment performance. Those diverge for reasons no split catches.
Reach for something else instead
- Cross-validation on small datasets — more reliable estimates by rotating the fold.
- Time-based backtesting for temporal problems, which mirrors how the model will actually be used.
- A live holdout or shadow deployment — the only measure that reflects reality, and the only one that catches distribution shift.
Sources & further reading
- Recht et al. (2019), Do ImageNet Classifiers Generalize to ImageNet? — accuracy drops on a fresh test set built the same way. The most important result here.
- Kaufman et al. (2012), Leakage in Data Mining — the ways information crosses the boundary without anyone noticing.
- Cawley & Talbot (2010), On Over-fitting in Model Selection and Subsequent Selection Bias — why tuning on the test set invalidates it, formally.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Fitting a scaler or imputer before splitting, leaking test-set statistics into training.
- Checking the test set repeatedly during development. After enough looks, it's a validation set and you have no test set.
- Assuming a good test score predicts production. It predicts performance on data drawn like your test data, which is a smaller claim than it sounds.
At a glance
FieldMachine Learning
Core ideahold data back to measure generalisation
Three-waytrain, validation, test
Touch the test setonce
Temporal datasplit by date
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Validation set vs. test set — the one you tune against vs. the one you're only allowed to look at once.