What is a confidence threshold?
A confidence threshold is a rule that converts a model’s confidence score into an action. Predictions scoring at or above the threshold are treated as trustworthy and acted on. Predictions below it are declined, flagged, or handed to a person. For a binary classifier, the threshold is the point at which a probability becomes a “yes.”
Moving the threshold trades precision against recall. A high threshold means fewer predictions are accepted but more of them are right. A low threshold accepts more predictions and lets more errors through. There is no neutral setting. The default of 0.5 in most software is an arbitrary choice that reflects nothing about the business.
Confidence thresholds in practice
A health insurer uses a model to classify incoming prior authorization requests as routine or needing clinical review. Approving a request that needed review is costly and sometimes harmful. Sending a routine request to a nurse is merely slow. The team sets the threshold for automatic approval high, accepts that a larger share of requests goes to the review queue, and tunes the threshold using the cost of each kind of error instead of a round number.
The threshold should be reviewed whenever the model is retrained, because retraining shifts the distribution of scores, and whenever the cost of errors changes. In Kirk Tech Solutions engagements, the threshold is treated as a business setting with a named owner, documented alongside the approval workflow it feeds, and kept out of the code.