What is automation bias?
Automation bias is the tendency of people to trust an automated system’s output over their own judgment and the evidence in front of them. It shows up in two ways. Errors of commission happen when a person acts on a wrong recommendation. Errors of omission happen when a person fails to act because the system raised no alert. Both come from the same habit: once a tool is usually right, checking it starts to feel like wasted effort, and the checking stops.
The effect was first studied with aircraft crews and clinicians, and it applies directly to AI. A reviewer who approves two hundred model-generated invoice matches a day stops reading the two hundredth. A generated summary that sounds confident gets forwarded without a glance at the source document.
Why automation bias matters
Automation bias defeats the control that most AI deployments rely on. Companies put a person in the loop so that a human catches the model’s mistakes, then measure success by how fast that person clears the queue. Under those incentives the review becomes a formality, and the company carries the liability of a human decision with the error rate of an unchecked model. The EU AI Act names the problem directly: Article 14 requires that people overseeing a high-risk system be able to remain aware of automation bias.
Automation bias in practice
A property and casualty insurer uses a model to flag claims for fraud review. Adjusters approve unflagged claims quickly and scrutinize flagged ones. Over a year, unflagged claims with obvious inconsistencies slip through, because the absence of a flag reads as a clearance. The fix is procedural. A random sample of unflagged claims gets a full review, adjusters see the specific reasons for each flag rather than a single score, and approval throughput stops being the only metric the team tracks.
What to watch for
- Show the evidence behind a recommendation, since a bare score invites rubber-stamping.
- Audit a sample of approved outputs independently to measure how often reviewers catch planted or known errors.
- Avoid rewarding reviewers only for speed, which trains them to agree with the system.
- Rotate or limit high-volume review sessions, because attention falls with repetition.
- Design the interface so that disagreeing with the model takes no more effort than agreeing with it.