Training Data
The examples a model learns from — where almost all of its capability and almost all of its failures come from, and the part of the work nobody wants to do.
Reviewed July 13, 2026Stable
When not to use it
- (It's the input, not a technique. The question is when to distrust it.)*
- Without looking at it. A hundred rows by hand. You will find something.
- Assuming your benchmark labels are right. ImageNet's test set is ~6% wrong.
- With train/test duplication. The most common cause of a score that's too good.
- Believing more data always helps. Past a quality threshold, filtering beats volume.
Reach for something else instead
- (Ways to need less of it.)*
- Transfer learning — start from a model that already learned the general thing.
- Data augmentation — more examples from the ones you have.
- Synthetic data — with the collapse caveats.
- Fixing labels — usually a bigger win than gathering more.
This entry is part of a longer guide: What is machine learning?
Read more on the blog
- I spent a month writing down what every AI concept depends on256 concepts, 350 prerequisite links, one rule. I wrote down my predictions first so I could not quietly retrofit the results. Two of them were wrong, and those are the interesting part.
- The model was right. Acting on it would have killed people.A pneumonia model learned that asthma lowers your risk of dying. It was correct about the data and dangerously wrong as guidance, and almost nothing in machine learning is built to tell the difference.
- AI bias and fairness: why 'fair' has no single answerAI now helps decide who gets a loan, an interview, bail, or medical priority, and the fear is that it does so unfairly. The instinct is to remove the bias and make the model fair. But a mathematical result makes that impossible in a precise way: several reasonable definitions of fairness cannot all hold at once, so fairness is not a bug to fix but a choice among competing values.
- Why AI works worse in your languageThe gap between English and everything else is not about linguistic difficulty. It is about training data share, tokenizer fitting, and where instruction-tuning stopped, and only one of those three is expensive to fix.
Further reading
- Sambasivan et al. (2021), "Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI — 92% of practitioners hit them. The most important applied-ML paper most people haven't read.
- Northcutt, Athalye & Mueller (2021), Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks — 3.3% average error; correcting it changes model rankings.
- Lee et al. (2022), Deduplicating Training Data Makes Language Models Better — and memorisation tracks duplication, which connects data to copyright.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Tuning the model before looking at the data. It's the wrong order and it's the common one.
- Treating benchmark labels as ground truth. They're 3.3% wrong on average.
- Not checking for train/test duplication.
- Assuming label errors are random. They're systematic, and they're concentrated on the hard cases.
At a glance
FieldMachine Learning
Why it mattersthe model is a compression of it; every gap is its gap
The findingdata cascades; 92% of practitioners hit them
Benchmark label errors3.3% average, ImageNet ~6%
Memorisation tracksduplication
The ceilingyour labels
DifficultyBeginner
Often compared with
More data vs. a better model — the second gets the attention and the first more often works. Try doubling the data before doubling the parameters.
Where this sits
A starting point. Nothing needs to come before it.
0Levelstarting point
0Needs firstnothing
227Opens up73% of 310
11Areascrosses fields
Computed from the prerequisite graph, not assigned. How this works