What is ensemble learning?
Ensemble learning trains several models and combines their outputs into one prediction. Each model makes partly different mistakes, so averaging or voting cancels many of them. The result is usually more accurate and more stable than the best single model.
Three approaches dominate. Bagging (bootstrap aggregating) trains many copies of one model on random samples of the data and averages them, which mainly reduces variance. A random forest applies bagging to decision trees and adds randomness to each split. Boosting trains models in sequence, with each new model correcting the errors left by the ones before it, which mainly reduces bias. Gradient-boosted decision trees and XGBoost work this way. Stacking feeds the predictions of different model types into a final model that learns how much to trust each one.
Ensemble learning in practice
A property insurer estimates the final cost of each new water-damage claim to set its reserves. A gradient-boosted model captures interactions among claim details, while a regression model handles claims outside the range the trees have seen. Blending the two with weights chosen on held-out data gives estimates that miss by less, and swing less from month to month, than either model alone.
What to watch for
- The gain comes from models that err differently, so combining near-identical models adds little.
- An ensemble is harder to explain than one model, so plan for tools such as Shapley values where decisions must be justified.
- Every member model must be monitored, retrained, and served, which adds operating effort.