Machine Learning

Accuracy

Accuracy is the share of a classifier's predictions that were correct, counting all classes together, which makes it easy to understand and misleading whenever one class is rare.

What is accuracy?

Accuracy is the number of correct predictions divided by the total number of predictions. From the confusion matrix, it is the true positives plus the true negatives, divided by everything. It is the first metric most people reach for, and the one most vendors quote, because it needs no explanation.

Its weakness is that it treats every record and every kind of error as equal. When one class is far more common than the other, which is the normal situation in fraud, failures, churn, and disputes, a model that always predicts the common class scores high accuracy while finding nothing. The null error rate, which is the accuracy of that do-nothing model, is the number any claimed accuracy must beat.

Why accuracy matters

Accuracy is a fair summary when the classes are roughly balanced and both kinds of mistake cost about the same. Sorting incoming documents at a law firm into a dozen types of similar frequency is such a case, and a single accuracy figure tells the intake manager what to expect. Screening transactions for fraud at a credit union is the opposite case. There, precision and recall on the fraud class describe what the model does, and accuracy describes almost nothing.

When a vendor reports accuracy, the useful follow-up questions are what the class balance of the test set was, what the null error rate was, and what the confusion matrix looked like. A model whose accuracy is below the null error rate is worse than no model at all. Classifier performance covers the fuller set of measures.

← Back to AI Glossary