Data Drift

The full account

The model didn't break. The world did.

A deployed model degrades and nothing in the code changed. No bad release, no corrupted weights, no failing test. The model that was accurate in March is quietly wrong in July, and the reason is not in the model at all — it's that the world the model learned no longer matches the world it's now scoring. This is data drift, and it is the default fate of every model left running long enough. Static models in a moving world decay; the only question is how fast and whether you're watching.

The trap is that drift is invisible from inside the system. Accuracy metrics need labels, and in production labels arrive late or never — you find out a fraud model missed something when the chargeback lands weeks later, not at inference time. So a drifting model keeps returning confident predictions, the dashboards stay green because nobody's computing accuracy in real time, and the degradation is discovered by a human noticing the outputs have gotten strange. By then it's been wrong for a while.

The three things people call "drift"

The word covers three distinct failures with different fixes, and conflating them is why teams reach for the wrong remedy.

Covariate shift — the inputs change while the underlying relationship holds. Your users skew younger, a new region comes online, a sensor is replaced with one that reads slightly differently. The mapping from features to target is still valid; the model just hasn't seen inputs like these. Often fixable by retraining on recent data.

Label drift — the distribution of the target itself moves. Fraud rates rise, churn climbs, the base rate the model calibrated to is no longer the base rate. A model tuned to a 2% positive rate misbehaves at 8% even if nothing else changed.

Concept drift — the actual relationship between inputs and target changes. The same input now implies a different outcome. This is the hardest kind: a spending pattern that meant "safe" in one economy means "risk" in another. Retraining helps only if you've noticed, because the historical data now teaches the wrong lesson.

The reason the distinction matters: covariate shift you can often detect without labels, by watching the inputs. Concept drift you frequently cannot detect without labels at all, because the inputs look normal and only the outcomes betray it.

Why monitoring the inputs is the whole game

Since labels are late, the practical discipline is to monitor what you can see immediately: the inputs and the model's own outputs. If the distribution of incoming features moves away from the training distribution, that's an early warning that fires before accuracy has visibly collapsed — statistical distances between the training and live feature distributions, tracked per feature, flagged when they cross a threshold. It won't catch pure concept drift, where inputs look fine, but it catches the common cases early and cheaply.

The mature version pairs input monitoring with delayed-label evaluation: watch the inputs for a fast, imperfect signal, and compute true accuracy whenever labels eventually arrive for a slow, honest one. The combination is what separates teams who catch drift in days from teams who catch it in quarters.

There is no "done"

The deepest lesson of drift is that a deployed model is not a finished artefact; it's a claim about the world that expires. The engineering question is never "is the model accurate" but "is the model still accurate", and answering it requires infrastructure that most projects build only after being burned once. Retraining cadence, drift alarms, a held-back stream of freshly-labelled data, a rollback path — these are the unglamorous machinery of a model that stays alive. Skipping them doesn't mean the model won't drift. It means you'll find out from a user.