Automation Bias
The tendency to trust an automated system's output more than the evidence in front of you, and to stop checking because it is usually right.
Reviewed July 16, 2026Stable
When not to use it
- As a reason to remove human review. The finding is that unmeasured review is weak, not that review is worthless.
- To explain any disagreement with a model. Sometimes the human is simply right and the system is wrong.
- As a property of the model. It is a property of the human-machine pair and has to be measured as one.
Reach for something else instead
- Unaided baseline testing — measure people without the system to establish what assistance actually changes.
- Selective assistance — surface the system only on cases where it adds measured value, rather than on everything.
- -
Read more on the blog
- Horizon: the law presumed the computer was rightHundreds prosecuted on the output of an accounting system later found not to be robust. No machine learning was involved, which is precisely why it belongs in this record.
- Williams v Detroit: the match was not the failureThe first publicly reported wrongful arrest from a face recognition match. The settlement's remedy is procedural, and it names a failure that has nothing to do with model accuracy.
- Human in the loop is weaker than it soundsWhen the AI was wrong, experienced radiologists went from 82% accurate to 45.5%. A review step is only a control if the reviewer sometimes disagrees, and almost nobody measures how often they do.
- Nine cases, and none was fixed by a better modelTerritory 6 closes. Nine documented cases, three containing no AI at all, and not one where the remedy that worked was a more accurate system.
Further reading
- Parasuraman & Riley (1997), Humans and Automation: Use, Misuse, Disuse, Abuse — the paper that named the failure modes.
- Geirhos et al. (2020), Shortcut Learning in Deep Neural Networks — why systems fail on the variation a reviewer is least primed to catch.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating "a human approves it" as a control without measuring how often approval differs from the system's recommendation.
- Assuming expertise confers immunity. It does not; the effect is documented in experienced professionals.
- Optimising alert sensitivity without regard to precision, which trades commission errors for disuse.
At a glance
FieldSafety & Ethics
NamedParasuraman and Riley, 1997
Two modescommission and omission
Mirror problemdisuse from false alarms
Moderatoraccountability
DifficultyBeginner → Intermediate
Where this sits
A starting point. Nothing needs to come before it.
0Levelstarting point
0Needs firstnothing
0Opens upnothing further
1Areastays here
Computed from the prerequisite graph, not assigned. How this works