What is holdout data?
Holdout data is the part of a dataset deliberately kept out of model training, so the model can be judged on examples it has never seen. Validation data and test data are both forms of holdout data. In everyday use, the term usually means the final untouched set checked just before deployment.
The simplest holdout is a random sample of rows. When records have a time order, a stronger choice is an out-of-time holdout, which trains on older records and holds out the most recent period. That mirrors how the model will be used once deployed. Either way, the holdout needs enough of the outcomes that matter, such as defaults or fraud cases, for its score to be reliable.
Holdout data in practice
A collections agency builds a model to rank newly placed accounts by likelihood of payment. A random holdout gives an impressive score, partly through information leakage, because the same debtors appear on both sides of the split. The team rebuilds the holdout from the most recent six months of placements and keeps all of a debtor’s accounts on one side. The score drops, and the new figure matches what the model goes on to achieve in production.
Marketing teams use the term for a related idea. A holdout group is a set of customers deliberately left out of a campaign, so that the campaign’s effect can be measured against them.