What is A/B testing for models?
A/B testing of models is a controlled experiment in which live traffic is randomly split between two model versions so their results can be compared on real cases at the same time. Each user, account, or transaction is assigned to one version, the test runs for a fixed period, and a predefined metric decides the winner.
Offline evaluation measures accuracy on historical data. An A/B test measures the business outcome, such as orders placed, payments collected, or tickets resolved, which is what the company actually cares about. The two can disagree, because a more accurate model does not always change behavior as expected.
A/B testing in practice
An online retailer tests a new recommendation model on a share of sessions for three weeks. The team measures revenue per session rather than click-through, since the earlier model generated many clicks that led nowhere. Assignment is by customer rather than by session, so the same shopper does not see both models on different visits.
What to watch for
- Fix the metric, sample size, and end date in advance, and resist stopping early when one side looks ahead.
- Assign at the right unit so one customer or account is never in both groups.
- Account for seasonality, since a test that spans a holiday or a month-end can favor one version by chance.
- In credit, insurance, or hiring decisions, check with compliance before treating similar people differently, and consider shadow mode or champion-challenger testing instead.