Training Data
The examples a model learns from — where almost all of its capability and almost all of its failures come from, and the part of the work nobody wants to do.
When not to use it
- (It's the input, not a technique. The question is when to distrust it.)*
- Without looking at it. A hundred rows by hand. You will find something.
- Assuming your benchmark labels are right. ImageNet's test set is ~6% wrong.
- With train/test duplication. The most common cause of a score that's too good.
- Believing more data always helps. Past a quality threshold, filtering beats volume.
Reach for something else instead
- (Ways to need less of it.)*
- Transfer learning — start from a model that already learned the general thing.
- Data augmentation — more examples from the ones you have.
- Synthetic data — with the collapse caveats.
- Fixing labels — usually a bigger win than gathering more.
Sources & further reading
- Sambasivan et al. (2021), "Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI — 92% of practitioners hit them. The most important applied-ML paper most people haven't read.
- Northcutt, Athalye & Mueller (2021), Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks — 3.3% average error; correcting it changes model rankings.
- Lee et al. (2022), Deduplicating Training Data Makes Language Models Better — and memorisation tracks duplication, which connects data to copyright.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Tuning the model before looking at the data. It's the wrong order and it's the common one.
- Treating benchmark labels as ground truth. They're 3.3% wrong on average.
- Not checking for train/test duplication.
- Assuming label errors are random. They're systematic, and they're concentrated on the hard cases.
At a glance
FieldMachine Learning
Why it mattersthe model is a compression of it; every gap is its gap
The findingdata cascades; 92% of practitioners hit them
Benchmark label errors3.3% average, ImageNet ~6%
Memorisation tracksduplication
The ceilingyour labels
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
More data vs. a better model — the second gets the attention and the first more often works. Try doubling the data before doubling the parameters.