What is validation data?
Validation data is a portion of labeled data kept out of training and used during development to make choices about a model. The team trains candidate models on the training data, scores each one on the validation data, and keeps whatever scores best. The choices include the model type, the features, and the hyperparameters. Validation scores also decide when to stop training a neural network, a technique called early stopping that limits overfitting.
Because every choice is made by looking at validation scores, the winning model’s validation score runs slightly optimistic. Separate test data, untouched until the end, gives an honest final estimate.
Validation data in practice
A city’s 311 center trains a model to route service requests to the right department. Supervisors label a few thousand recent requests, which are set aside as validation data. Three model types are compared on that set. The results also show the model confusing streetlight outages with traffic signal faults, which leads the city to tighten its category definitions. Only the chosen model is then scored on the test set.
What to watch for
- Some vendors and research fields swap the terms validation and test, so ask which set produced any reported score.
- Draw validation data from the same period and population the model will serve.
- When data is scarce, cross-validation rotates the validation role across the whole dataset instead of fixing one split.