What is cognitive bias?
A cognitive bias is a systematic shortcut in human thinking that produces predictable errors in judgment. Confirmation bias favors evidence that supports an existing belief. Anchoring gives too much weight to the first number seen. Availability bias overrates events that are easy to recall. These are features of how people think under time pressure, and every person has them.
Cognitive bias reaches AI systems from two directions. Training data records past human decisions, so a model trained on historical hiring or underwriting outcomes learns the biases of the people who made them. And the people who use a model bring their own biases to its output, anchoring on a suggested price or accepting a confident answer without verifying it, which is the pattern called automation bias.
Why cognitive bias matters
A company evaluating an AI tool is itself subject to these biases. A demo that goes well anchors expectations. Early successes are remembered and early failures explained away. A sound evaluation defines the test set and the pass criteria before anyone sees results, and includes cases chosen to break the system.
In a logistics firm, dispatchers who review a routing model’s suggestions tend to accept them during busy periods and override them only when a route looks unusual. Measuring accepted and overridden routes against actual delivery times shows which habit serves the business, and the answer is often different from what the dispatchers believe.