What is Cohen’s kappa?
Cohen’s kappa is a statistic that measures agreement between two raters who sort the same items into categories, after removing the agreement expected by chance. The formula is observed agreement minus chance agreement, divided by one minus chance agreement, where chance agreement comes from how often each rater uses each label. A kappa of 1 means perfect agreement, 0 means no better than chance, and negative values mean worse than chance. Published scales differ, but most treat values above 0.8 as strong agreement and values around 0.5 as moderate at best.
In machine learning, kappa is a common check on data annotation, showing whether labels are consistent enough to train on. It can also score a model against ground truth with chance removed, which helps when one class dominates. Weighted kappa handles ordered categories such as severity levels, and Fleiss’ kappa extends the idea to more than two raters.
Cohen’s kappa in practice
A corporate legal department asked two contract managers to flag supplier contracts with a non-standard indemnity clause, as training data for a review model. They agreed on 180 of 200 contracts. Most contracts had no such clause, so chance alone would have produced agreement near 0.80, and kappa came out at 0.49. The team wrote a sharper definition with examples, relabeled a sample, and reached a kappa above 0.8 before training. Consistent labels matter because a model scored against inconsistent ones is marked wrong on some cases it got right.