Cross-Validation
Testing on every part of your data by rotating which part you hold back — the fix for "my score depends on which rows I happened to set aside."
Reviewed July 11, 2026Stable
When not to use it
- When you have plenty of data. A single large held-out set is a fine estimate and k times cheaper.
- When training is expensive. Five folds means five training runs. On a large model that's a real budget decision, not a formality.
- On time series, in its naive form. Random folds let the model learn from the future. Use forward-chaining splits.
- As a substitute for a genuinely held-out final test. If you tuned against your folds, you need untouched data to estimate honestly.
Reach for something else instead
- A single train/validation/test split — simpler, sufficient with enough data.
- Nested cross-validation — when you must both select and estimate. Correct, and expensive.
- Time-series forward chaining — the only honest option for temporal data.
- Bootstrap — resampling with replacement; different bias/variance trade-off.
Cross-validation rotates which slice is used for testing.
Instead of one train/test split (whose score depends on luck of the draw), k-fold cross-validation splits the data into k parts and trains k times, each time holding out a different part for testing. Averaging the k scores gives a far more reliable estimate of real-world performance — and reveals whether the model is consistent or just got lucky on one split.
Further reading
- Stone (1974), Cross-Validatory Choice and Assessment of Statistical Predictions — the foundational treatment.
- Kohavi (1995), A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection — the empirical study behind "use 10-fold stratified", still the practical reference.
- Cawley & Talbot (2010), On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation — why selecting and estimating on the same folds inflates your score.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Fitting the scaler or imputer before splitting. Leakage, inflated score, and it looks like nothing is wrong.
- Reporting the mean and hiding the spread, which is where the real information was.
- Not stratifying on imbalanced classification, producing folds where the minority class barely appears.
- Not grouping when rows share an entity — the same patient in train and test is memorisation scored as generalisation.
- Selecting hyperparameters and reporting the winning fold-average as an unbiased estimate. It isn't; you selected on it.
At a glance
FieldMachine Learning
What it fixesscore depends on which rows you held back
Standard5- or 10-fold, stratified
Most informative outputthe variance, not the mean
Main hazardpipeline leakage
DifficultyBeginner
Often compared with
Cross-validation vs. a single train/test split — the same idea, done k times. Worth it when data is scarce or the decision matters; wasteful when data is plentiful.
Where this sits
3 concepts come first. Understanding it opens up 4 more.
3Levelsteps in
3Needs firstconcepts
4Opens up1% of 310
2Areascrosses fields
Learn these firstTrain/Test Split
Computed from the prerequisite graph, not assigned. How this works