Train/Test Split
Holding back data the model never sees, so you can find out whether it learned anything or just memorised.
When not to use it
- Random splitting on time-series data. You'd be training on the future. Split by date, without exception.
- Random splitting when rows are grouped. Multiple records per customer across the boundary means the model learns the customer, not the pattern.
- As your only evidence. A clean split measures held-out performance, not deployment performance. Those diverge for reasons no split catches.
Reach for something else instead
- Cross-validation on small datasets — more reliable estimates by rotating the fold.
- Time-based backtesting for temporal problems, which mirrors how the model will actually be used.
- A live holdout or shadow deployment — the only measure that reflects reality, and the only one that catches distribution shift.
Read more on the blog
- How to tell if your AI actually worksMost teams ship AI features on vibes, then argue about whether changes helped. An evaluation set is an afternoon of work and it settles every argument you're about to have.
- How to read a model releaseEvery few weeks a lab announces a new frontier model and every headline says the same thing. Here's how to work out what actually changed, what the benchmark numbers mean, and which parts of the announcement are marketing.
After enough looks, it is a validation set and you have no test set.
Every model here is worthless by construction: the labels are coin flips, so nothing is learnable and the honest accuracy of anything is 50%. Each “model” is one random guess evaluated on a held-out set. Try one and you get 50%, give or take. Try forty, keep the best, and you will report something in the sixties — not because it is better, but because you looked forty times at the same 200 points and kept whichever noise flattered you. Nothing was cheated and no rule was broken; each individual evaluation was legitimate. The test set simply stopped being one somewhere around the fifth look.
The full account
The rule is trivial. Following it is not.
Hold out data, don't train on it, evaluate on it once. There is nothing to understand. It is taught in the first week of every course and it is the single most violated rule in applied machine learning — not by people who don't know it, but by people who know it perfectly well and broke it anyway without noticing.
The reason is that the split is a claim about information, not about rows in a dataframe. The test set must contain nothing the model could have learned from. And information leaks through channels that look nothing like copying rows: through a preprocessing step fitted before the split, through a duplicate record, through a decision you made after looking at the score, through time. Each of those is invisible in the code. All of them produce the same symptom, which is a number that is too good and a model that fails in production.
How big is this actually
Kapoor and Narayanan tried to measure it in 2023, and the answer is worse than the folklore. Surveying the literature across fields that had adopted ML, they found leakage documented in 17 distinct scientific fields, affecting at least 294 papers — the published Patterns figure; their earlier preprint counted 329 — in some cases producing what they call wildly overoptimistic conclusions. These are not undergraduate assignments. They are peer-reviewed publications in medicine, genomics, political science, neuroimaging.
Their case study is the part that should stop you. Civil war prediction had a small literature claiming complex ML models substantially outperformed logistic regression — a result with a clean story attached, that ML finds structure classical statistics misses. Kapoor and Narayanan corrected the leakage and re-ran it. The advantage vanished. The complex models did not perform substantively better than a decades-old logistic regression. The entire published finding was an artefact of information crossing the split.
Sit with the shape of that. The claim was not that ML was slightly oversold. It was that in a field that had adopted ML on the strength of a measured improvement, the improvement was not there, and nobody noticed for years because everyone was measuring it the same wrong way.
The eight ways it gets in
Their taxonomy is worth knowing because the failures are so much more mundane than the drama suggests. They range, in their phrase, from textbook errors to open research problems:
No test set at all — the model was evaluated on data it trained on, sometimes by accident of a pipeline. Preprocessing on the full dataset — the most common competent-person error. You fit the scaler, the imputer, or the feature selector before splitting. The scaler now knows the test set's mean. The model was trained with a whisper of the answer. Feature selection on the full dataset — same mechanism, larger effect: you chose which features matter using data that includes the test set. Duplicates across the split — the same patient, the same near-identical image, the same reposted sentence, on both sides. Illegitimate features — a column that encodes the target. The hospital's treatment_started field predicts diagnosis beautifully and does not exist at prediction time. Temporal leakage — training on the future to predict the past. Any random split of a time series does this by construction. Non-independence between train and test — same subject, same site, same device on both sides, so the model learns the subject rather than the disease. Sampling the test set differently from the population you'll deploy on.
Notice how few of those look like cheating. Most look like a tidy pipeline.
The leak that has no fix
The figure above demonstrates a different channel, and it is the one you cannot engineer around. It does not require touching the test set with code. It requires only looking at the score.
Try forty model variants. Keep the best one on the test set. That best-of-forty result is not an unbiased estimate of anything — it is the maximum of forty noisy draws, and the maximum of noisy draws is biased upward by construction. The figure computes it live, over 140 simulated runs, using the expected best-of-N rather than a lucky seed: a best-of-40 score of 57.5% corresponds to a true quality around 50.1%. You selected the noise. The test set is now, quietly, part of your training loop — not because you trained on it, but because you chose on it.
Every hyperparameter search does this. Every leaderboard does this at community scale, with thousands of teams as the forty variants. Recht and colleagues tested the community version directly in 2019: they built a brand-new ImageNet test set from the same distribution, following the original collection procedure as closely as they could, and every model dropped — with the ranking largely preserved, which suggests the field had been fitting the specific test set rather than the task, and doing it collectively over a decade.
| Leak | Looks like | Fix |
|---|---|---|
| Preprocessing before split | tidy code | fit inside the fold, always |
| Duplicates across split | a big dataset | dedup on content, not ID |
| Temporal | a normal random split | split by time, never at random |
| Non-independence | more data | group split by subject/site/device |
| Selecting on the test score | doing your job | a third holdout, touched once |
What honest practice looks like
Split first, before you touch anything. Every preprocessing step goes inside the pipeline, fitted on the training fold only, and you get this for free by using a pipeline object rather than a sequence of transformations.
Split by time if there is time, by group if there are groups, and at random only if you have positively established that neither applies. The random split is the default and the default is wrong more often than it is right.
Keep a third set that you touch exactly once, at the end, and treat every number you produced while iterating as what it is — a training signal. And when you report the final figure, report how many variants you tried to get there, because that number determines how much of your result is the maximum of noise.
Finally: the discipline scales with consequence, not with sophistication. Kapoor and Narayanan's 294 papers were not written by careless people. They were written by domain experts using standard tools correctly, in fields where nobody was checking, because the split looks like a solved problem and its violations do not look like violations.
Further reading
- Recht et al. (2019), Do ImageNet Classifiers Generalize to ImageNet? — accuracy drops on a fresh test set built the same way. The most important result here.
- Kaufman et al. (2012), Leakage in Data Mining — the ways information crosses the boundary without anyone noticing.
- Cawley & Talbot (2010), On Over-fitting in Model Selection and Subsequent Selection Bias — why tuning on the test set invalidates it, formally.
- Kapoor & Narayanan (2023), Leakage and the Reproducibility Crisis in Machine-Learning-Based Science — Patterns; leakage found in 294 papers across 17 fields, with a taxonomy of eight types.
- Kapoor & Narayanan (2021), Claims of Superior Performance of Machine Learning over Logistic Regression for Civil War Prediction Don't Reproduce — the case study: fix the leakage, and the ML advantage disappears entirely.
- Recht, Roelofs, Schmidt & Shankar (2019), Do ImageNet Classifiers Generalize to ImageNet? — build a fresh test set from the same distribution; every model drops.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Fitting a scaler or imputer before splitting, leaking test-set statistics into training.
- Checking the test set repeatedly during development. After enough looks, it's a validation set and you have no test set.
- Assuming a good test score predicts production. It predicts performance on data drawn like your test data, which is a smaller claim than it sounds.
At a glance
Often compared with
Where this sits
2 concepts come first. Understanding it opens up 26 more.
Computed from the prerequisite graph, not assigned. How this works