Model Monitoring
Watching a deployed model for the failures that don't raise errors — and the thing you most need to watch is the thing you can't see.
Reviewed July 13, 2026Stable
When not to use it
- (Monitoring choices that mislead.)*
- Input drift as a proxy for accuracy. It fires when nothing matters and misses what does.
- Statistical significance as an alert threshold. At scale everything is significant. Use effect size.
- Aggregate metrics only. Your model can be fine overall and broken for one segment entirely.
- Standard APM alone. It was built for systems that crash. This one returns 200 and lies.
Reach for something else instead
- Prediction drift — the best signal per unit of effort, and most teams lack it.
- Labelled sampling — slow, expensive, the only thing that measures truth.
- Conformal prediction — coverage guarantees, no labels needed. Underused.
- Shadow deployment — compare against the incumbent on real traffic before switching.
Read more on the blog
- Where AI has not landed: 77% report no use caseTransportation reports 7.5% AI use, construction 9.5%, against 73% for large information-sector firms. The most common reason given is not cost or skills. It is that no use case applies.
- 621,000 robots installed, virtually no humanoidsTerritory 7 opens on the question article 117 left unresolved. Industrial robotics is enormous and growing. The general-purpose machine that would move the boundary has almost no deployments.
- 2.6 million robotic surgeries, none of them autonomousThe most-deployed medical robot makes no decisions. And the largest meta-analysis of its outcomes was co-authored by the company that sells it.
- Every chatbot query on earth is 2% of AI's powerTerritory 8 opens on the numbers behind AI's physical footprint, and on the arithmetic in the IEA's own report that almost nobody quotes.
Further reading
- Breck et al. (2017), The ML Test Score — the monitoring axis; most teams score near zero.
- Rabanser, Günnemann & Lipton (2019), Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift — test the model's representation, not raw features.
- Angelopoulos & Bates (2023), Conformal Prediction: A Gentle Introduction — coverage guarantees without labels or distributional assumptions.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Expecting failures to be loud. It returns 200 with a confident wrong answer.
- Monitoring inputs and calling it monitoring. That's a hint, not a measurement.
- Alerting on p-values at scale. You'll drown, then you'll mute it.
- Not noticing your alerts are caused by your responses to alerts. Retraining shifts the distribution that triggers the monitor.
At a glance
FieldApplied AI
The problemmodels fail silently and return 200
The central difficultyaccuracy needs labels; labels arrive late or never
The best cheap signalprediction drift, at the output
The rulealert on effect size, never significance
The unsolved caseLLMs; all proxies
DifficultyIntermediate
Often compared with
Input drift vs. prediction drift — one fires when features move whether or not it matters; the other only moves when something reached the output. Watch the second.
Where this sits
8 concepts come first. Understanding it opens up 1 more.
5Levelsteps in
8Needs firstconcepts
1Opens up0% of 310
1Areastays here
Computed from the prerequisite graph, not assigned. How this works