Why AI models get worse: forgetting, collapse, and drift

A model that worked last quarter is worse now. Nobody changed anything, or someone changed one thing, and accuracy is quietly sliding. When this happens, three different explanations get reached for, usually interchangeably: the model forgot, the model collapsed, the data drifted.

These are three different mechanisms. They have different causes, different fingerprints, and different fixes, and misdiagnosing which one you have is how teams spend a month retraining a model whose only problem was that the world changed underneath it. Here's how to tell them apart.

Silent failure is the normal case

Worth stating plainly before the mechanisms, because it shapes how you should think about all of them: none of these failures announces itself.

A model that has drifted does not raise an error. It returns a confident answer that is wrong slightly more often than it used to be, and the difference is invisible in any individual response. The first signal is usually not a monitoring alert but a support ticket, a complaint from a team that stopped trusting the output months ago, or a quarterly review where someone notices the numbers moved.

This is different from how most software fails. A service that breaks throws an exception, a page that breaks renders wrong, and both are noticed within minutes. A model that degrades keeps working, keeps returning plausible output, and keeps being believed. The failure mode is quiet, and quietness is what makes it expensive: by the time it surfaces, the wrong outputs have been acted on.

The practical consequence is that you cannot rely on discovering this reactively. Detection has to be built in advance, because the system will not tell you.

The fourth cause nobody looks for

Three mechanisms are covered below. There is a fourth that is more common than any of them and is almost never diagnosed as model degradation, because it is not.

The model did not change. The inputs did.

An upstream system starts sending a field in a different format. A logging change alters how text is truncated. A vendor updates their API and a value that was previously always present becomes optional. A team fixes a data-quality bug that the model had quietly learned to depend on. In each case the model performs exactly as it always has, on inputs that no longer look like what it was trained on.

This is worth checking first because it is the cheapest to rule out and the fastest to fix. Compare a sample of current inputs against a sample from training, field by field, looking at nulls, ranges, formats and category distributions. If something moved, you have your answer without touching the model at all.

The reason it gets misdiagnosed is that the symptom is identical to genuine drift. Output quality falls. The difference is that drift means the world changed and the relationship the model learned no longer holds, while this means the pipeline changed and the model is being fed something different. One requires retraining; the other requires a pipeline fix and takes an afternoon.

Catastrophic forgetting: the model learned something new and lost something old

Neural networks don't file new knowledge next to old knowledge. Learning means adjusting weights, and the weights that encode a new task are the same weights that encoded the previous one. Train a network on task A, then train it on task B, and its performance on A doesn't gently fade. It can fall off a cliff. Psychologists McCloskey and Cohen documented this in 1989, long before deep learning, and named it catastrophic forgetting: the new learning overwrites the old.

This is why it shows up exactly when teams do the responsible-sounding thing: fine-tune the model on new data. Fine-tune a general model hard on your domain, and it gets better at your domain while quietly getting worse at things it used to handle, instruction following, other topics, edge cases nobody re-tested. The failure is invisible until someone exercises the old capability, because nothing in the new training run measures what was lost.

The research response has a representative example in elastic weight consolidation, Kirkpatrick and colleagues' 2017 method that identifies which weights mattered most for old tasks and makes them stiffer, so new learning routes around them. It helps, and the broader family of continual-learning methods helps, but none of it repeals the underlying fact: in a shared-weight system, learning is overwriting, managed rather than eliminated.

The fingerprint: performance dropped on old capabilities immediately after a training event, a fine-tune, a continued pretraining run, an update. If nobody trained anything, this isn't your problem.

Model collapse: the training data was already an echo

Model collapse is what happens when models train on the output of models. Shumailov and colleagues showed it in a peer-reviewed Nature paper in 2024: train a model, generate data from it, train the next model on that data, repeat, and the models degrade generation over generation. The rare and unusual disappears first, because generated data under-represents the tails of the real distribution; keep iterating and the outputs converge toward a bland, low-variance center. They called the stages early collapse (the tails vanish) and late collapse (the distribution shrivels).

The result is real, and the mechanism matters for anyone building on a web that increasingly contains AI-generated text. But the headline version, the internet is poisoning itself and future models are doomed, overstates what the paper showed, and a follow-up made the boundary precise. Gerstgrasser and colleagues asked whether collapse is inevitable and found that it depends on a detail the doom narrative skips: collapse arises when each generation's synthetic data replaces the real data. When synthetic data accumulates alongside the original real data, which is closer to how actual training corpora evolve, collapse was avoided across the model sizes and architectures they tested. Real data acts as an anchor. The lesson isn't "synthetic data is poison"; it's "don't throw away the human data, and know what fraction of your corpus is an echo."

The fingerprint: this is a training-pipeline disease, not a deployment one. It shows up as shrinking diversity, outputs getting samey, rare cases handled worse each release, in systems whose training data includes generated content. If your model is frozen and untouched, it cannot be collapsing.

Data drift vs concept drift: the distinction that decides your fix

These two get used as synonyms and they are different failures requiring different responses. Getting the diagnosis wrong means retraining when you should be re-labelling, or vice versa.

Data drift is a change in the inputs. The distribution of what arrives shifts, while the underlying relationship between input and outcome stays the same. Your fraud model was trained mostly on desktop transactions and traffic moved to mobile. The model still knows what fraud looks like; it is now seeing a population it saw less of during training.

Concept drift is a change in the relationship itself. The inputs may look identical while what they mean has changed. The same transaction pattern that indicated fraud last year is now ordinary behaviour, because fraudsters adapted or because customer habits moved. The model's knowledge is not incomplete, it is out of date.

The asymmetry in that table is the important part. Data drift can be detected without knowing whether you were right. Comparing the statistical shape of today's inputs against training inputs needs no labels, which is why input monitoring is the cheapest early-warning system available.

Concept drift cannot be caught that way. The inputs can look completely stable while the relationship underneath them rots, and the only signal is that your predictions stopped matching outcomes. That requires labels, which arrive late if they arrive at all, which is why concept drift is usually discovered by the business rather than by the monitoring.

A practical rule: if input monitoring is quiet and accuracy is falling, suspect concept drift. If input monitoring is noisy, check whether it is data drift or the pipeline change described above, because those produce identical alerts.

Data drift: the model is fine, the world moved

The most common of the three needs no exotic mechanism at all. A model is a snapshot of the relationship between inputs and outcomes at training time. Deploy it, and the world keeps moving: customer behaviour shifts, vocabulary changes, fraudsters adapt precisely because your model caught last year's pattern, a pandemic rewrites what "normal purchasing" looks like overnight. The model didn't change. Its assumptions expired. That's data drift, and its sharper cousin, concept drift, where the same inputs start meaning different outcomes.

Drift is the degradation mode that requires no one to touch anything, which is what makes it the default suspect for any slow, steady decline in a deployed system. It's also the one with the most mature toolkit: monitor the input distribution against a training-time baseline, monitor prediction confidence, and, where labels eventually arrive, monitor actual accuracy over time. This is the bread and butter of model monitoring, and the teams that treat it as plumbing rather than an afterthought are the ones whose incident reports say "detected in week one" instead of "customers noticed."

The fingerprint: gradual decline in a deployed model with no training events, often seasonal or tracking a real-world change you can name. Check the input statistics before blaming the model, usually the data will confess.

What detection actually requires

Given that none of this is self-announcing, the question is what you have to build.

A held-out reference set that does not change. A fixed sample of inputs with known-good outputs, run on a schedule, with results tracked over time. This is the single highest-value thing on the list because it converts an invisible problem into a line on a chart. The set has to be frozen; if you refresh it, you lose the ability to compare across time, which was the entire point.

Input distribution monitoring. Track the statistical shape of what arrives: the distribution of each feature, the rate of nulls, the frequency of each category. This catches the pipeline problem above and gives early warning of genuine drift, usually before output quality moves enough to notice.

Output distribution monitoring. Track the shape of what leaves. If a classifier that predicted the positive class four percent of the time starts predicting it eleven percent of the time, something changed even if you cannot yet say what. Output monitoring needs no labels, which is why it is often the only monitoring that exists.

A labelled trickle. Some proportion of live predictions checked by a human, continuously rather than in campaigns. This is the only thing that measures actual accuracy rather than proxies for it, and it is the thing most often cut when budgets tighten, which is a false economy since everything else is inference from indirect signals.

The ordering matters. Teams commonly start with the sophisticated statistical drift tests and never build the fixed reference set, which is backwards. The reference set is simpler, cheaper, and catches more.

The diagnosis table

Three mechanisms, one symptom, and the fix for each is the other two's waste of time. Retraining cures drift and causes forgetting if done carelessly. Adding data cures forgetting and causes collapse if the data is an echo. The order of operations that respects all three: monitor for drift continuously, retrain deliberately with old-capability evals in place, and know the provenance of every batch you train on.

The uncomfortable summary is that degradation is the default state of a deployed model, not the exception. Systems don't stay good; they're kept good, by teams who can tell these three apart.

The short version

Models can seem to get worse after deployment for several distinct reasons that are easy to confuse. Drift is the world changing while the model stays fixed, making its knowledge stale. Catastrophic forgetting is a model losing old abilities when updated or fine-tuned, because learning new patterns overwrites shared weights. Model collapse is quality eroding across generations when models train on data produced by earlier models rather than humans. These have different causes and different fixes: retraining on fresh data helps drift but can worsen forgetting or collapse, and it is expensive. The first step is diagnosis, telling the three apart, because the wrong fix can make things worse. A model's weights do not rot on their own; what degrades is the fit between a fixed model and a changing world, plus the side effects of how we retrain, and each cause needs its own remedy.

Common questions

What is the difference between data drift and concept drift? Data drift is a change in the input distribution while the input-to-outcome relationship holds: the model still knows what it is looking for and is now seeing a population it saw less of in training. Concept drift is a change in the relationship itself: the same inputs now mean something different, so the model's knowledge is out of date rather than incomplete. The practical difference is detectability. Data drift can be spotted from inputs alone with no labels, while concept drift is invisible in the inputs and only surfaces when predictions stop matching outcomes.

How do I detect data drift? Compare the statistical shape of current inputs against a frozen sample from training: the distribution of each feature, the rate of nulls, the frequency of each category. This requires no labels, which makes it the cheapest monitoring available and usually the earliest warning you will get. Watch for the confound described above, since a pipeline change produces identical alerts to genuine drift and is far quicker to fix.

Which is more dangerous, data drift or concept drift? Concept drift, because it is silent. Data drift announces itself in the inputs, which you can monitor continuously without labels. Concept drift leaves the inputs looking normal while the relationship underneath changes, so the only signal is degraded accuracy, which requires labelled outcomes that typically arrive weeks or months later. By the time it is detectable, the wrong predictions have already been acted upon.

How do I tell forgetting, collapse, and drift apart? By what changed. Catastrophic forgetting follows a training event and hits old capabilities; model collapse shows up as shrinking output diversity in systems trained on generated data; data drift is a gradual decline in a deployed model that nobody touched, the world moved. Check the fingerprint before choosing a fix.

Is the internet "poisoning itself" with AI-generated data? Overstated. Model collapse happens when synthetic data replaces real data across generations; when synthetic data accumulates alongside the original human data, closer to how real corpora evolve, the research found collapse was avoided. Real data acts as an anchor.

Why is retraining not a universal fix? Because the three mechanisms have opposite cures. Retraining cures drift but can cause forgetting; adding data cures forgetting but can cause collapse if the data is an echo. The fix for one is the other's waste of time, which is why diagnosis has to come first.

Why would an AI model get worse over time? A deployed model can appear to degrade for several distinct reasons. The world changes while the model stays fixed, so its knowledge becomes stale and its answers drift out of date, which is drift. Updates or fine-tuning can cause it to lose earlier abilities, known as catastrophic forgetting. And when models are trained on data increasingly produced by other models, quality can degrade across generations, sometimes called model collapse. The weights do not rot on their own; what changes is the fit between a fixed model and a moving world, or the effect of retraining and data quality on successive versions.

What is model collapse? Model collapse is the degradation that can occur when models are trained on data generated by earlier models rather than by humans. Each generation slightly narrows the diversity of the data, losing rare cases and tail behaviour, and training on that thinner distribution produces a model that is itself thinner, which then generates even less diverse data for the next round. Over successive generations, quality and variety can erode. It is a concern as AI-generated text fills the web and gets scraped into future training sets, though careful data curation and mixing in human data can mitigate it.

What is catastrophic forgetting? Catastrophic forgetting is when training a model on new information causes it to lose abilities it previously had. Because a neural network stores what it knows in shared weights, adjusting those weights to learn a new task can overwrite the patterns that supported old ones. It is a common pitfall in fine-tuning: teaching a model a narrow new skill can quietly degrade its general capabilities. Techniques exist to reduce it, such as mixing old and new data or limiting how much the weights move, but it is a fundamental reason that updating a model is not simply additive and must be done carefully.

Does retraining fix a model that has gotten worse? Not universally, because the different causes need different fixes. Retraining on fresh data addresses drift, where knowledge has gone stale, and can restore currency. But retraining does not automatically solve catastrophic forgetting, which can be reintroduced by the very update, or model collapse, which retraining on more model-generated data can worsen. Retraining is also expensive and risks changing behaviour in unintended ways. The right response depends on which problem you actually have, which is why distinguishing drift, forgetting, and collapse matters before reaching for a retrain.