What is champion-challenger testing?
Champion-challenger testing is a method for improving a production model by running one or more candidate models, the challengers, alongside the current model, the champion, and promoting a challenger when it proves better. The champion continues to make the real decisions. The challengers score the same cases, either silently in shadow mode or on a small share of live traffic, and their predictions are compared with the champion’s once outcomes are known.
Unlike a one-off A/B test, champion-challenger testing is a standing program with a champion always in production, a queue of challengers, and promotion criteria set in advance. The approach is common in credit risk, fraud detection, and collections, where outcomes arrive weeks after the prediction.
Champion-challenger testing in practice
A regional bank runs a fraud model on card transactions. Two challengers score every transaction in shadow mode, and their alerts are recorded without being acted on. Chargebacks arrive over the following weeks and become the ground truth. After a quarter, one challenger shows higher precision at the same recall, meaning fewer blocked legitimate purchases for the same fraud caught. The model risk committee promotes it, and the old champion becomes a challenger.
What to watch for
- Set the promotion criteria and the review period before the challenger starts.
- Score challengers on exactly the cases the champion saw, with the same features and timing, or the comparison is invalid.
- Keep the comparison records, since they document why the production model changed.