Collapse needs you to throw the old data away

TL;DR. Shumailov and colleagues showed in Nature in 2024 that training successive generations of a model on the previous generation's output degrades it, losing the tails of the distribution first and low-frequency events permanently. The effect held across variational autoencoders, Gaussian mixtures and language models, and in one experiment produced perplexity increases of 20 to 28 points. The condition, stated in the paper, is that each generation's data replaces the last. Gerstgrasser and colleagues asked what happens when data accumulate instead, which is what actually occurs on the open web, and found collapse avoided empirically across three model families and proved analytically that test error has a finite upper bound independent of the number of iterations. The popular claim drops the condition that produces the result. And the question is still open, because under accumulation the real-data fraction falls toward zero anyway.

---

Status: established with conditions, and actively disputed. Primary sources: Shumailov et al., Nature 2024; Gerstgrasser et al., arXiv:2404.01413; Kazdan et al. on extending the accumulate setting; and papers arguing the opposite, including one titled Model Collapse Does Not Mean What You Think. The literature is genuinely unsettled and this article does not resolve it.

---

What the Nature paper found

Train a generative model. Sample from it. Train the next model on those samples. Repeat.

The distribution degrades in two phases. Early collapse: distributional errors accumulate and the model drifts from the true distribution. Late collapse: low-frequency events disappear permanently, because a sample of finite size rarely contains the tails, and what is not sampled cannot be learned.

The effect was demonstrated across model families, on variational autoencoders, Gaussian mixture models and a language model, which is what makes it a statistical result rather than an artefact of one architecture.

In the language experiment, five epochs of training with no real data retained produced perplexity increases of 20 to 28 points.

And the collapse is not gradual in a way that early testing catches. It can look mild for several generations and then accelerate, which means a practitioner who tried one round and saw nothing has evidence about one round.

All of that is correct and the paper states its condition: data are replaced at each iteration.

What happens if data accumulate

Gerstgrasser and colleagues asked the obvious follow-up. In the real world, synthetic text does not delete the web that preceded it. It is added to it.

They describe their own setting as maximally pessimistic: a hypothetical future in which synthetic data are uncontrollably dumped onto the internet and vacuumed up for the next model.

Empirically, accumulation avoided collapse across every model family they tested: transformers on causal language modelling, diffusion models on molecular conformation, and variational autoencoders on images. Each generation trains on the original real data plus everything generated since.

And they proved it analytically. In the tractable linear framework where earlier work showed test error rising with each iteration under replacement, accumulation gives a finite upper bound on test error, independent of the number of iterations.

Later work extended the result. Fine-tuning a 2-billion-parameter model on an instruction dataset collapses under replacement and does not under accumulation, and related work identifies three loop types, fully synthetic, augmented with human data, and accumulating, of which only the last two survive with sufficient human fraction.

Which is the whole difference

Replace: the proportion of real data becomes zero immediately after the first iteration.

Accumulate: the proportion of real data falls asymptotically toward zero and is never zero at any finite step.

That is the entire distinction between a result that diverges and one that is bounded, and it is the sentence most often missing when the finding is described.

The popular version, that AI trained on AI degrades, is not wrong so much as unconditioned. It reports the conclusion without the assumption that produced it, and the assumption is the part that determines whether it applies to anything.

This is citation decay in its cleanest scientific form: not a fabricated number, not a misattribution, but a load-bearing condition falling off during transmission until a bounded finding reads as an inevitability.

And the question is not settled

A debunk would stop here and it would be wrong to.

One paper argues explicitly that the accumulation result is weaker than it appears, because under accumulation the real-data fraction still tends to zero, and cites work concluding that collapse cannot generally be mitigated unless a strategy asymptotically removes all but a vanishing proportion of synthetic data.

Another argues the phenomenon is a statistical property of repeated fitting and sampling that may be unavoidable in principle, independent of how the data are managed.

And the field itself is messy. One survey of this literature lists roughly eighteen papers using different experimental methodologies and different mathematical assumptions about different generative models, reaching different conclusions, and states plainly that this makes assessing the probability and harm of collapse difficult.

So the honest position is three-part. The Nature result is correct under replacement. The accumulation result is correct and is closer to real conditions. And whether either describes what actually happens to models trained on the web after 2023 is not established by any of them, because none of them is measuring that.

What nobody has measured

The empirical question is what fraction of training data is now model-generated and whether it matters.

No published figure exists that anyone should trust. Estimates of synthetic content prevalence on the web vary enormously, depend entirely on detection methods with poor and unstable accuracy, and are produced mostly by parties selling detection.

And the labs that would know do not say. They know their data mixtures, their filtering pipelines and their deduplication. None of that is disclosed, which is the pattern from the previous territory: the party with the complete data has no obligation to publish and does not.

Which means the practically important version of this question has no evidence at all, while the theoretical version has a substantial and contradictory literature.

Three things this establishes

A condition is part of a finding, not a caveat on it. Replacement against accumulation is the difference between divergence and a bounded error, proved formally. A result quoted without its condition is not a weaker version of the result. It is a different claim.

Analytic and empirical agreement is the strong case. Gerstgrasser and colleagues found the same answer in experiments across three model families and in a proof, which is a much better evidential position than either alone, and it is worth noting that the popular narrative survived it.

And a contested literature is not the same as an unknown one. Eighteen papers disagreeing is a signal that assumptions differ, not that nothing is known. The known part is the conditional structure. The unknown part is which condition the world is in.

What it does not establish

That collapse cannot happen. Under replacement it demonstrably does, and closed loops where a system trains on its own filtered output are a real deployment pattern.

That accumulation is safe indefinitely. The bound is finite and it is not zero, the real-data fraction still falls, and the papers disputing the significance of that are making a substantive argument rather than a pedantic one.

That web data is accumulating cleanly. Deduplication, filtering and curation all change the effective mixture in ways the theoretical framework does not model.

And nothing about current models. No result here describes a system anyone has shipped, because the relevant data mixtures are undisclosed.

What is unresolved

Whether the real-data fraction tending to zero matters. This is the live technical disagreement and it is not close to settled.

What fraction of the web is model-generated. No trustworthy figure exists, detection is unreliable, and the estimates in circulation come mostly from vendors of detection.

Whether curation changes the picture entirely. Real training pipelines filter aggressively, and a filtered accumulation is neither of the two regimes that have been analysed.

And whether any of it is observable from outside. The labs know their mixtures. Nobody else does, and no obligation exists to change that.

The counter-argument

Emphasising the condition may understate a real risk. Closed-loop training on a model's own output is a common pattern in practice, including self-distillation and synthetic instruction data, and in those settings the replacement regime is not a hypothetical. A reader who takes from this that collapse is a myth has taken the wrong lesson.

The accumulation proof is in a linear framework. It is analytically tractable precisely because it is simplified, and extending a linear result to transformers trained on web-scale corpora requires assumptions that the empirical work supports and does not establish.

The real-data fraction objection may be decisive. If the proportion of genuinely human data tends to zero, a finite error bound may still describe a model that is bounded away from the thing it was meant to learn. That is an argument this article reports and does not adjudicate, and it may be the one that ends up mattering.

And treating this as a citation decay story is convenient. The corpus has a concept for it, the case fits neatly, and fitting a case to a concept is exactly the failure the corpus warns about elsewhere.

The short version

Shumailov and colleagues showed in Nature that training generations of a model on the previous generation's output degrades the distribution, losing tails early and low-frequency events permanently, across variational autoencoders, Gaussian mixtures and language models, with perplexity rising 20 to 28 points in one experiment.

The stated condition is that each generation's data replaces the last.

Gerstgrasser and colleagues asked what happens when data accumulate, which is what the open web does, described their own setting as maximally pessimistic, and found collapse avoided empirically across transformers, diffusion models and autoencoders, with an analytic proof that test error has a finite upper bound independent of the number of iterations.

Replace and the real-data proportion is zero after one step. Accumulate and it falls asymptotically toward zero and is never zero. That difference is the entire result, and it is the sentence that goes missing.

The popular claim is not false. It is unconditioned, which makes it a different claim from the one that was proved.

And the question remains open. One paper argues the accumulation result is weaker than it appears because the real-data fraction still vanishes. Another argues the effect is an unavoidable property of repeated fitting. A survey of the area lists around eighteen papers with different methods and assumptions reaching different conclusions.

Meanwhile the practically important question, what fraction of training data is now model-generated and whether it matters, has no trustworthy measurement at all, because detection is unreliable and the labs that know their own mixtures do not publish them.

Common questions

What did the Nature paper actually show? That training successive generations of a generative model on the previous generation's samples degrades the learned distribution, in two phases: early collapse, where distributional errors accumulate and the model drifts, and late collapse, where low-frequency events disappear permanently because finite samples rarely contain the tails. It held across variational autoencoders, Gaussian mixture models and a language model, and in one language experiment five epochs with no real data retained produced perplexity increases of 20 to 28 points.

What is the condition people leave out? That the data are replaced at each iteration, so each generation trains only on its predecessor's output. That assumption is in the paper and it is what produces divergence. Without it the result does not follow.

What happens if data accumulate instead? Collapse is avoided. Gerstgrasser and colleagues tested accumulation across transformers on language, diffusion models on molecular conformation and variational autoencoders on images, and found no collapse in any of them. They also proved analytically, in the tractable linear framework where replacement makes test error grow with each iteration, that accumulation gives a finite upper bound on test error independent of the number of iterations.

Why does that difference matter so much? Because under replacement the proportion of real data is zero immediately after the first iteration, while under accumulation it falls asymptotically toward zero and is never zero at any finite step. That is the difference between a quantity that diverges and one that is bounded, and it is the whole result.

So is model collapse a myth? No, and that would be the wrong lesson. Under replacement it demonstrably occurs, and closed-loop training on a model's own output is a real deployment pattern including self-distillation and synthetic instruction data. The narrow claim is that the popular version drops the condition that produces the result, which makes it a different claim rather than a simplified one.

Is the accumulation result the final word? No. One paper argues it is weaker than it appears because the real-data fraction still tends to zero under accumulation, citing work concluding that collapse cannot generally be mitigated unless a strategy removes all but a vanishing proportion of synthetic data. Another argues the phenomenon is an unavoidable statistical property of repeated fitting and sampling. A survey of this literature lists roughly eighteen papers using different methods and assumptions and reaching different conclusions.

How much of the web is actually model-generated? Nobody knows, and no figure in circulation should be trusted. Detection methods have poor and unstable accuracy, and most published estimates come from parties selling detection. The organisations that do know their own training mixtures, filtering and deduplication are the labs, and none of them publishes that information.

What should a practitioner take from this? That the regime matters more than the phenomenon. If a pipeline trains on its own output and discards the original data, the replacement result applies and the risk is real and can appear suddenly after looking mild for several generations. If real data is retained and synthetic data is added alongside it, the accumulation result applies and the evidence, both empirical and analytic, says the error stays bounded.