What is regularization?
Regularization is a set of techniques that limit how complex a model can become during training. The training process normally tries to minimize error on the training data alone. Regularization adds a penalty for complexity to the loss function, so the model must balance fitting the data against staying simple. The result generalizes better to records it has never seen.
In a linear model, the penalty applies to the size of the coefficients. Ridge regression shrinks all coefficients toward zero. Lasso regression can push some of them to exactly zero, which removes those features entirely. In neural networks, dropout randomly disables a fraction of the connections during each training pass, and weight decay does a similar shrinking job. Early stopping, which halts training once validation error stops improving, is regularization by another route.
Why regularization matters
Business datasets are small and noisy compared with the datasets used to train large models. A model that is free to grow as complex as it likes will find patterns in the noise and will fail on next quarter’s data. Regularization is the main defense against this overfitting, and the strength of the penalty is one of the hyperparameters that model tuning adjusts.
Regularization also makes models easier to explain. A credit risk model at a regional bank that uses twelve features after lasso regularization is easier to document for examiners than one that uses two hundred, and its feature importance values are more stable from one retraining to the next.
Regularization in practice
An insurance carrier builds a model to predict which claims will need an adjuster visit. The first version, trained without a penalty, scores well on historical claims and poorly on the holdout data. The team adds a ridge penalty and tunes its strength with cross-validation. Training accuracy drops slightly, holdout accuracy rises, and the gap between the two closes. That gap is the number worth tracking.
What to watch for
- Set the penalty strength with validation data, because a penalty that is too strong causes underfitting.
- Normalize features before applying a penalty, since the penalty treats a coefficient on dollars differently from one on counts.
- Ask whether a vendor’s reported results came from a model tuned on data separate from the test set.
- Retune the penalty when the training data grows, because larger datasets usually need less regularization.