Machine Learning

Null Error Rate

The null error rate is the share of predictions that would be wrong if a model always predicted the most common class. A model's error rate must fall below it before its accuracy shows any value.

What is the null error rate?

The null error rate is the share of predictions that would be wrong if a model ignored its inputs and always predicted the most common class. If 12 of every 100 scheduled clinic appointments end in a no-show, a model that predicts every patient will attend is wrong 12 times in 100, a null error rate of 0.12. In a two-class problem, it equals the prevalence of the minority class. With several classes, it is one minus the share of the largest class.

It is the error rate of the simplest possible baseline model. Statistical software sometimes reports its complement, the accuracy of that baseline, as the no-information rate.

Why the null error rate matters

An accuracy figure means little until it is set against the null error rate. A vendor tells a health system that its no-show model is 0.89 accurate. Predicting that everyone attends is already 0.88 accurate, so the model improves on doing nothing by one prediction in a hundred. The scheduling team needs to know how many real no-shows the model catches and how many of its flags are right, because those patients get reminder calls or have their slots overbooked.

A model whose error rate exceeds the null error rate makes more mistakes than doing nothing, although it can still earn its place by catching rare and costly cases. Ask vendors to state the null error rate of their test set next to any accuracy they quote.

← Back to AI Glossary