What is cross-validation?
Cross-validation is a method for estimating how well a model will perform on new data by training and testing it several times on different slices of one dataset. In k-fold cross-validation, the data is divided into k equal parts called folds, commonly five or ten. The model is trained on all folds but one and scored on the fold left out. The cycle repeats until every fold has been scored once. The average score estimates performance, and the spread between folds shows how stable that estimate is.
Every record serves in both training and evaluation, which makes the most of a small dataset. The cost is k training runs instead of one. Teams use cross-validation to compare models and tune hyperparameters, then retrain the chosen model on all the data.
Cross-validation in practice
A machine shop has records for 600 production runs, each marked pass or fail at final inspection, with only a few dozen failures. A single holdout would test on a handful of failures, so the score would depend on which ones landed there. Five-fold cross-validation scores the model on every run, and stratified folds spread the failures evenly across the five.
What to watch for
- Time-ordered data needs folds that always train on earlier periods and test on later ones.
- Keep all records for one customer, patient, or machine in the same fold, or the score will be inflated.
- Fit preparation steps such as scaling on the training folds only, so no information leaks from the test fold.