What is test data?
Test data is a set of labeled examples kept apart from everything used to build a model and scored once at the end. Training data teaches the model, and validation data guides the choices made along the way. The test set answers a single question: how well the finished model is likely to perform on cases it has never seen. It should resemble production data, drawn from the same population and, when patterns change over time, from the most recent period.
Test data in practice
A litigation team uses a classifier to rank several hundred thousand emails by how likely each is to be responsive to a discovery request. Before relying on the ranking, attorneys code a random sample of documents that played no part in training. The team measures recall on that sample, meaning the share of responsive documents the classifier catches at the chosen cutoff. That figure is reported under the review protocol agreed with opposing counsel.
What to watch for
- Adjusting the model after seeing test results makes the test score optimistic, so an honest estimate then needs a fresh test set.
- Make sure near-duplicate records, such as several emails from one thread, do not appear in both training and test data, a common form of information leakage.
- For a generative AI assistant, build a test set from real questions with approved answers, because public benchmarks may have leaked into the model’s training data.