What is generalization?
Generalization is how well a trained model performs on new records drawn from the same kind of data it was trained on. A model that generalizes well has learned the underlying pattern. A model that generalizes poorly has memorized its training set, which is overfitting, or never learned the pattern at all, which is underfitting.
Generalization is measured by holding data back. The test data and holdout data are kept away from training and tuning, and the model’s score on them is the estimate of how it will behave in production. The estimate is only as good as the holdout’s resemblance to real future data.
Why generalization matters
Every business use of a model is a bet on generalization. A county permit office that trains a model to route incoming applications to the right department will run it on next year’s applications, which will mention new projects, new street names, and new form versions. Performance on last year’s data is only useful as a predictor of performance on those.
Generalization also has limits that a score cannot show. A model trained on one county’s forms may not generalize to a neighboring county with a different form layout, even if it scores well on its own holdout set. When the world changes, as in data drift or concept drift, a model that generalized well at launch stops doing so. That is why model monitoring continues after deployment.