What is classifier performance?
Classifier performance is how well a classification model assigns inputs to the right class, measured on test data the model did not train on. Every measure starts from the confusion matrix, which counts true positives, false positives, true negatives, and false negatives. The standard metrics are ratios of those four counts.
| Metric | What it answers | When it is the right measure |
|---|---|---|
| Accuracy | What share of all predictions were correct | Classes are balanced and both errors cost the same |
| Precision | Of the records flagged positive, how many really were | Acting on a false alarm is expensive |
| Recall | Of the real positives, how many were caught | Missing a case is expensive |
| F1 score | The balance of precision and recall in one number | Both errors matter and classes are imbalanced |
| AUC | How well the model ranks positives above negatives across all thresholds | Comparing models before a threshold is chosen |
Why classifier performance matters
The metric chosen decides which model wins and how it is tuned, so it must reflect the cost of each kind of error in the business. A model that screens transactions for anti-money laundering review is judged on recall, because a missed case is a regulatory failure, and the price is a large volume of false positives for analysts to clear. A model that decides which marketing emails to send is judged on precision, because every wrong send has a small cost and a missed one has almost none. Accuracy alone misleads whenever one class is rare, which is the usual case.
Classifier performance in practice
A managed service provider builds a model to flag tickets that will breach their service-level agreement. The first version reports high accuracy, but almost no tickets breach, so the model achieves that by flagging nothing. Switching the objective to recall on the breach class, with precision tracked alongside, produces a model that catches most breaches at the cost of a review queue the dispatcher can handle. The team then picks the confidence threshold that keeps the queue within capacity.
What to watch for
- Report performance per class, because a strong overall number can hide a class the model never gets right.
- Always ask what the prevalence of the positive class was in the test set.
- Compare any claimed result against the null error rate of predicting the majority class.
- Check calibration separately, since a model can rank well and still report misleading confidence scores.