What is calibration?
Calibration is the agreement between what a model says its confidence is and how often it is actually right. A calibrated model that assigns a score of 0.8 to a thousand predictions should be correct on roughly eight hundred of them. If it is correct on nine hundred, it is underconfident. If it is correct on six hundred, it is overconfident. Calibration is checked by grouping predictions into score bands and comparing the predicted rate with the observed rate in each band, often drawn as a reliability curve.
Accuracy and calibration are independent. A model can rank cases in the right order and still report scores that are systematically too high. Gradient-boosted trees and modern neural networks are often overconfident, and techniques that correct class imbalance make this worse. Post-processing methods such as Platt scaling and isotonic regression adjust the scores after training without changing the ranking.
Why calibration matters
Every business rule built on a confidence threshold assumes the scores mean what they say. A credit union that auto-approves small loans when the model’s default probability is below 0.05 is relying on that number being honest. If the model is overconfident, the actual default rate in the auto-approved group will be higher than the policy intended, and nobody will see it until the losses arrive.
Calibration also drifts. A model that was calibrated at launch can become overconfident as the population changes, so the reliability check belongs in model monitoring alongside accuracy.