Distribution Shift
When the data a model meets in use no longer resembles the data it learned from, and its accuracy falls without anything in the model changing.
Reviewed July 16, 2026Stable
When not to use it
- To explain every performance drop. Pipeline faults and label quality changes look identical from the outside and are more common.
- As a reason to retrain on a schedule. Retraining on a temporary shift degrades the model when conditions revert.
- Where the deployment population was never the training population to begin with; that is a validation failure, not drift.
Reach for something else instead
- Domain adaptation — explicitly train for a target distribution you know differs from the source.
- Selective prediction — let the model abstain when inputs fall outside what it has seen, rather than answering anyway.
- -
Read more on the blog
- Why AI aces the test and fails the variationTransformers scored 96 to 99% on a semantic parsing benchmark and 16 to 35% on the same task with the pieces recombined. The gap has a name, a 35-year argument behind it, and a fix that suggests the capability was there all along.
- Why AI models get worse: forgetting, collapse, and driftThree different mechanisms quietly degrade AI systems, catastrophic forgetting, model collapse, and data drift. They get blamed for each other constantly. Here's how to tell them apart, and which one is actually eating your accuracy.
- 621,000 robots installed, virtually no humanoidsTerritory 7 opens on the question article 117 left unresolved. Industrial robotics is enormous and growing. The general-purpose machine that would move the boundary has almost no deployments.
- 220 million miles, inside a boundary Waymo drewThe strongest safety evidence in physical autonomy, and the methodology that makes it honest is also what limits what it can tell you.
Further reading
- Recht et al. (2019), Do ImageNet Classifiers Generalize to ImageNet? — a new test set by the original protocol, and accuracy fell across the board.
- Geirhos et al. (2020), Shortcut Learning in Deep Neural Networks — why shift exposes shortcuts the training distribution concealed.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Monitoring accuracy alone, which lags the shift by however long outcomes take to arrive.
- Treating a statistically significant distribution change as automatically consequential; most are not.
- Refreshing the training set without refreshing the test set, which hides the problem rather than fixing it.
At a glance
FieldMachine Learning
Three kindscovariate, label, concept
Detected viainput monitoring, confidence distribution
Consequencetest accuracy is an upper bound
Standard responseretrain, but not always
DifficultyIntermediate
Where this sits
A starting point. Nothing needs to come before it.
0Levelstarting point
0Needs firstnothing
0Opens upnothing further
1Areastays here
Computed from the prerequisite graph, not assigned. How this works