Cross-Validation
Testing on every part of your data by rotating which part you hold back — the fix for "my score depends on which rows I happened to set aside."
When not to use it
- When you have plenty of data. A single large held-out set is a fine estimate and k times cheaper.
- When training is expensive. Five folds means five training runs. On a large model that's a real budget decision, not a formality.
- On time series, in its naive form. Random folds let the model learn from the future. Use forward-chaining splits.
- As a substitute for a genuinely held-out final test. If you tuned against your folds, you need untouched data to estimate honestly.
Reach for something else instead
- A single train/validation/test split — simpler, sufficient with enough data.
- Nested cross-validation — when you must both select and estimate. Correct, and expensive.
- Time-series forward chaining — the only honest option for temporal data.
- Bootstrap — resampling with replacement; different bias/variance trade-off.
Sources & further reading
- Stone (1974), Cross-Validatory Choice and Assessment of Statistical Predictions — the foundational treatment.
- Kohavi (1995), A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection — the empirical study behind "use 10-fold stratified", still the practical reference.
- Cawley & Talbot (2010), On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation — why selecting and estimating on the same folds inflates your score.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Fitting the scaler or imputer before splitting. Leakage, inflated score, and it looks like nothing is wrong.
- Reporting the mean and hiding the spread, which is where the real information was.
- Not stratifying on imbalanced classification, producing folds where the minority class barely appears.
- Not grouping when rows share an entity — the same patient in train and test is memorisation scored as generalisation.
- Selecting hyperparameters and reporting the winning fold-average as an unbiased estimate. It isn't; you selected on it.
At a glance
FieldMachine Learning
What it fixesscore depends on which rows you held back
Standard5- or 10-fold, stratified
Most informative outputthe variance, not the mean
Main hazardpipeline leakage
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Cross-validation vs. a single train/test split — the same idea, done k times. Worth it when data is scarce or the decision matters; wasteful when data is plentiful.