Synthetic Data
Training on data a model generated — increasingly standard, genuinely useful, and carrying a failure mode with a Nature paper attached.
When not to use it
- As a replacement for real data. That's the recursive setup that collapses. Accumulate, don't replace.
- Without a verifier, at scale. Unfiltered synthetic data is the generator's beliefs, errors included.
- For rare events and tails. Those are the first thing the generator under-represents — exactly what you needed.
- Where the distribution matters and can't be checked. Judgement, taste, strategy. No filter, no signal.
Reach for something else instead
- Real data — the thing this substitutes for, with the caveats.
- Data augmentation — transformations of real data. Lower risk, less coverage.
- Rejection sampling with a verifier — the version that reliably works.
- Transfer learning — use a model that already saw real data.
Sources & further reading
- Shumailov et al. (2024), AI models collapse when trained on recursively generated data — Nature; the mechanism, and the tails go first.
- Gerstgrasser et al. (2024), Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data — the correction: accumulate, don't replace.
- Wang et al. (2022), Self-Instruct: Aligning Language Models with Self-Generated Instructions — how most open instruction data actually exists.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading model collapse as "synthetic data is doomed." Accumulating real plus synthetic avoids it; only replacement fails.
- Generating without filtering, then wondering why the model inherited the generator's errors.
- Expecting synthetic data to cover rare cases. Finite sampling loses the tails first — that's the mechanism.
- Assuming your web crawl is human-written. Increasingly it isn't, and nothing labels it.
At a glance
FieldMachine Learning
The failure modemodel collapse; the tails go first, then the variance
Whyfinite resampling under-represents rare events, compounding each generation
The fixaccumulate real + synthetic; don't replace
Where it genuinely worksverifiable domains with a filter
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Synthetic data with a verifier vs. without — one adds external signal and works indefinitely; the other recycles the generator's beliefs and degrades.