Automation Bias

Category: Probability & Belief

Your tendency to trust an automated system's output over your own judgment, and over contradictory evidence sitting right in front of you, treating "the computer said so" as the end of the inquiry instead of the start.

How it works

Automation bias runs on the same engine as most of your shortcuts: effort avoidance. A confident automated recommendation becomes a heuristic replacement for vigilant information seeking, which is the polite academic way of saying it gives you permission to stop thinking.

This produces two distinct failures. Commission errors happen when you actively do what the system tells you even though other evidence screams no. Omission errors happen when the system stays silent about a problem and you never catch it, because you outsourced the watching to it.

The cruel twist is that the more reliable the system usually is, the harder this bias bites. High reliability trains what researchers call "learned carelessness," so you are least prepared at the exact moment the dependable tool finally gets it wrong.

Where you'll see it

  • Skitka, Mosier, and Burdick (1999) ran glass-cockpit pilots through simulated flights with an automated monitoring aid. Pilots who had the aid caught fewer events on their own than pilots without it, and when the aid issued a false alarm, many complied against fully valid instrument readings. In related NASA cockpit work, pilots given a false automated cue shut down a perfectly healthy engine, despite insisting beforehand they never would.
  • Povyakalo and colleagues (2013) re-analyzed a computer-aided detection study in breast screening and found the software actively hurt the better radiologists. When the CAD prompts were wrong, cancers that skilled readers would have caught on their own got missed, because the readers deferred to the machine. Automation did not just fail to help the experts, it degraded them.
  • Goddard, Roudsari, and Wyatt (2012) reviewed automation bias in clinical decision-support systems and found that in prescribing tasks, doctors switched correct answers to incorrect ones about 5 percent of the time after a computer gave bad advice. Less experienced clinicians flipped their answers more often, which is the last thing you want from the person still learning the job.
  • In Mata v. Avianca (2023), two New York lawyers filed a brief citing six court cases that ChatGPT had invented. When challenged, they went back to ChatGPT, which confidently confirmed the fake cases were real and findable on Westlaw. Judge Castel fined them 5,000 dollars. The tool was wrong, and they trusted it twice.

Where it comes from

The term was coined by Kathleen Mosier and Linda Skitka in the mid-1990s out of NASA-funded aviation research, defined as the tendency to use automated cues as a heuristic replacement for vigilant information seeking and processing. Mosier, Skitka, Burdick, and Heers (1996) documented commission and omission errors in glass-cockpit pilots, and the follow-up "Does automation bias decision-making?" (Skitka, Mosier, and Burdick, 1999) established the effect experimentally. The concept later spread from cockpits into medicine, driving, and now generative AI, but the founding insight has held: a good automated aid can make you worse at the very task it is supposed to help with.

How to counter it

Demand a second independent source before acting. The output counts as one data point, not a conclusion. Before you follow it, confirm it against a channel the automation cannot see: a raw instrument, a primary document, a manual recalculation. In Mata v. Avianca, one search on Westlaw would have exposed all six fake cases.

Make yourself accountable out loud. Mosier's own studies found that pilots who felt personally responsible for verifying automation checked its output far more and made fewer errors. Say who has to defend this decision and how, before you rely on the machine. "I will have to explain this to the review board" beats "the system flagged it" every time.

Distrust the tool most when it has been perfect. A long streak of correct outputs is exactly what breeds learned carelessness. Build a fixed verification habit that does not relax with the system's track record, because reliability drops right when your guard does.

Watch the silences, not just the alerts. Omission errors come from the problems the system never mentions. Periodically ask "what would this tool fail to warn me about?" and go look for those things by hand.

The tell

You catch yourself explaining a decision with "well, the system said" or "the model flagged it," and you cannot state a single independent reason you checked yourself. If your justification is the tool's confidence rather than your own verification, the bias is already driving.

Related biases

References

  1. Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automation bias decision-making?. International Journal of Human-Computer Studies, 51(5), 991-1006
  2. Mosier, K. L., Skitka, L. J., Heers, S., & Burdick, M. (1998). Automation Bias: Decision Making and Performance in High-Tech Cockpits. The International Journal of Aviation Psychology, 8(1), 47-63
  3. Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121-127
  4. Parasuraman, R., & Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors, 52(3), 381-410
  5. Povyakalo, A. A., Alberdi, E., Strigini, L., & Ayton, P. (2013). How to Discriminate between Computer-Aided and Computer-Hindered Decisions: A Case Study in Mammography. Medical Decision Making, 33(1), 98-107