Why AI models get worse: forgetting, collapse, and drift

Three different mechanisms quietly degrade AI systems — catastrophic forgetting, model collapse, and data drift. They get blamed for each other constantly. Here's how to tell them apart, and which one is actually eating your accuracy.

A model that worked last quarter is worse now. Nobody changed anything — or someone changed one thing — and accuracy is quietly sliding. When this happens, three different explanations get reached for, usually interchangeably: the model forgot, the model collapsed, the data drifted.

These are three different mechanisms. They have different causes, different fingerprints, and different fixes — and misdiagnosing which one you have is how teams spend a month retraining a model whose only problem was that the world changed underneath it. Here's how to tell them apart.

Catastrophic forgetting: the model learned something new and lost something old

Neural networks don't file new knowledge next to old knowledge. Learning means adjusting weights, and the weights that encode a new task are the same weights that encoded the previous one. Train a network on task A, then train it on task B, and its performance on A doesn't gently fade — it can fall off a cliff. Psychologists McCloskey and Cohen documented this in 1989, long before deep learning, and named it catastrophic forgetting: the new learning overwrites the old.

This is why it shows up exactly when teams do the responsible-sounding thing: fine-tune the model on new data. Fine-tune a general model hard on your domain, and it gets better at your domain while quietly getting worse at things it used to handle — instruction following, other topics, edge cases nobody re-tested. The failure is invisible until someone exercises the old capability, because nothing in the new training run measures what was lost.

The research response has a representative example in elastic weight consolidation — Kirkpatrick and colleagues' 2017 method that identifies which weights mattered most for old tasks and makes them stiffer, so new learning routes around them. It helps, and the broader family of continual-learning methods helps, but none of it repeals the underlying fact: in a shared-weight system, learning is overwriting, managed rather than eliminated.

The fingerprint: performance dropped on old capabilities immediately after a training event — a fine-tune, a continued pretraining run, an update. If nobody trained anything, this isn't your problem.

Model collapse: the training data was already an echo

Model collapse is what happens when models train on the output of models. Shumailov and colleagues showed it in a peer-reviewed Nature paper in 2024: train a model, generate data from it, train the next model on that data, repeat — and the models degrade generation over generation. The rare and unusual disappears first, because generated data under-represents the tails of the real distribution; keep iterating and the outputs converge toward a bland, low-variance center. They called the stages early collapse (the tails vanish) and late collapse (the distribution shrivels).

The result is real, and the mechanism matters for anyone building on a web that increasingly contains AI-generated text. But the headline version — the internet is poisoning itself and future models are doomed — overstates what the paper showed, and a follow-up made the boundary precise. Gerstgrasser and colleagues asked whether collapse is inevitable and found that it depends on a detail the doom narrative skips: collapse arises when each generation's synthetic data replaces the real data. When synthetic data accumulates alongside the original real data — which is closer to how actual training corpora evolve — collapse was avoided across the model sizes and architectures they tested. Real data acts as an anchor. The lesson isn't "synthetic data is poison"; it's "don't throw away the human data, and know what fraction of your corpus is an echo."

The fingerprint: this is a training-pipeline disease, not a deployment one. It shows up as shrinking diversity — outputs getting samey, rare cases handled worse each release — in systems whose training data includes generated content. If your model is frozen and untouched, it cannot be collapsing.

Data drift: the model is fine, the world moved

The most common of the three needs no exotic mechanism at all. A model is a snapshot of the relationship between inputs and outcomes at training time. Deploy it, and the world keeps moving: customer behaviour shifts, vocabulary changes, fraudsters adapt precisely because your model caught last year's pattern, a pandemic rewrites what "normal purchasing" looks like overnight. The model didn't change. Its assumptions expired. That's data drift — and its sharper cousin, concept drift, where the same inputs start meaning different outcomes.

Drift is the degradation mode that requires no one to touch anything, which is what makes it the default suspect for any slow, steady decline in a deployed system. It's also the one with the most mature toolkit: monitor the input distribution against a training-time baseline, monitor prediction confidence, and — where labels eventually arrive — monitor actual accuracy over time. This is the bread and butter of model monitoring, and the teams that treat it as plumbing rather than an afterthought are the ones whose incident reports say "detected in week one" instead of "customers noticed."

The fingerprint: gradual decline in a deployed model with no training events, often seasonal or tracking a real-world change you can name. Check the input statistics before blaming the model — usually the data will confess.

The diagnosis table

	What changed	Typical trigger	First check
Forgetting	The weights	A fine-tune or update	Old-capability evals, before vs after
Collapse	The training data's provenance	Generated data in the corpus	Output diversity across releases
Drift	The world	Nothing — time passed	Input distribution vs training baseline

Three mechanisms, one symptom, and the fix for each is the other two's waste of time. Retraining cures drift and causes forgetting if done carelessly. Adding data cures forgetting and causes collapse if the data is an echo. The order of operations that respects all three: monitor for drift continuously, retrain deliberately with old-capability evals in place, and know the provenance of every batch you train on.

The uncomfortable summary is that degradation is the default state of a deployed model, not the exception. Systems don't stay good; they're kept good, by teams who can tell these three apart.

Common questions

How do I tell forgetting, collapse, and drift apart? By what changed. Catastrophic forgetting follows a training event and hits old capabilities; model collapse shows up as shrinking output diversity in systems trained on generated data; data drift is a gradual decline in a deployed model that nobody touched — the world moved. Check the fingerprint before choosing a fix.

Is the internet "poisoning itself" with AI-generated data? Overstated. Model collapse happens when synthetic data replaces real data across generations; when synthetic data accumulates alongside the original human data — closer to how real corpora evolve — the research found collapse was avoided. Real data acts as an anchor.

Why is retraining not a universal fix? Because the three mechanisms have opposite cures. Retraining cures drift but can cause forgetting; adding data cures forgetting but can cause collapse if the data is an echo. The fix for one is the other's waste of time — which is why diagnosis has to come first.