What is an ROC curve?
A receiver operating characteristic (ROC) curve is a chart that shows how a binary classifier trades detection against false alarms as its decision threshold moves. The vertical axis is the true positive rate, also called recall or sensitivity. The horizontal axis is the false positive rate, the share of actual negatives that the model wrongly flags. Each point on the curve represents one possible threshold. The name comes from radar engineering in the Second World War, where engineers measured how well receivers separated real aircraft from noise.
How the ROC curve works
The model gives every case in a labeled test set a score. A threshold above every score flags nothing and sits at the bottom-left corner of the chart. A threshold below every score flags everything and sits at the top-right corner. Sweeping the threshold between those extremes traces the curve. The diagonal between the two corners is what random guessing produces, and the further a curve bows toward the top-left corner, the better the model separates the classes. The area under the curve (AUC) condenses the whole curve into one number.
Both rates are calculated within a single actual class, so the curve does not change with prevalence. That makes it useful for comparing models across datasets, but it also hides how many false alarms a small false positive rate produces when negatives vastly outnumber positives. The precision-recall curve shows that problem directly.
ROC curve in practice
A regional property insurer compares its referral rules with a new model for sending claims to its special investigations unit. The rules have no adjustable score, so they appear as a single point on the chart, with a true positive rate of 0.55 at a false positive rate of 0.08. The model’s curve passes above that point. At the same false positive rate it reaches 0.75, and it matches the rules’ 0.55 at a false positive rate of 0.03. The claims director picks the point on the curve that produces a weekly referral volume the investigators can handle, then sets the model’s confidence threshold to match.
What to watch for
- Focus on the part of the curve where the business will operate, which is usually the far left where false positive rates are low.
- Check precision at the chosen threshold, because a curve can look excellent while most flagged cases are false alarms.
- Ask how many positive cases the curve was built from, since a handful of positives produces a jagged and unreliable curve.
- Ask how the positive labels were produced, because fraud in claims that were never investigated is never recorded and the curve cannot count it.