What is overfitting?
Overfitting is the failure mode in which a model memorizes the training data instead of learning the patterns behind it. The model fits the quirks and noise of the specific records it was trained on. It then scores very well on those records and much worse on records it has never seen. The gap between training performance and holdout performance is the signature of overfitting.
Any model can overfit, but the risk grows with model complexity and shrinks with the amount of training data. A decision tree allowed to grow until every leaf holds one record will reproduce the training set perfectly and predict poorly. A neural network with millions of parameters trained on a few thousand records will do the same.
Why overfitting matters
An overfit model looks excellent in the demo and disappoints in production, which is the worst possible order for a business decision. The money has been spent and the process has been changed before the weakness appears. Overfitting is also the usual explanation when a vendor’s reported accuracy cannot be reproduced on the customer’s own data.
Overfitting in practice
A regional trucking company builds a model to predict which loads will deliver late. The training data covers one year. The model learns that loads assigned to two specific drivers in February were late, because those drivers were out sick that month. The pattern is real in the training data and useless afterward. Cross-validation would have shown the model scoring far worse on February records held out of training.
The standard defenses are more training data, regularization, simpler models, early stopping, and removing features that carry noise rather than signal. Cross-validation and a clean holdout set detect the problem but do not prevent it.
What to watch for
- Insist on seeing performance on a holdout set that was never used for training or tuning.
- Treat a near-perfect training score as a warning sign rather than good news.
- Check that the holdout set comes from a later time period, because random splits hide overfitting to seasonal effects.
- Expect some decline in production performance even from a well-built model, and plan model monitoring to measure it.