What are gradient-boosted decision trees?
Gradient-boosted decision trees (GBDT) are a form of boosting that builds one model as the sum of many small decision trees. The first tree makes a rough prediction. The second is trained on what the first got wrong, the third on what the first two still get wrong, and so on for hundreds or thousands of rounds. Each tree is weak on its own, often only a few levels deep. Together they capture nonlinear patterns and interactions that a linear model misses.
How gradient-boosted decision trees work
The word gradient describes how each new tree’s target is chosen. The method uses a loss function that measures error, such as squared error for a forecast or log loss for a yes-or-no outcome. Each new tree is fitted to the negative gradient of that loss, the direction that reduces error fastest, which for squared error equals the remaining error on each record. A learning rate scales down each tree’s contribution, so the model improves in small steps. Training stops when performance on validation data stops improving, a practice called early stopping. Because the loss can be swapped, the same method handles regression, classification, ranking, and count forecasts. XGBoost, LightGBM, and CatBoost are the most widely used open-source implementations.
Gradient-boosted decision trees in practice
A collections agency scores every newly placed account for its likelihood of paying within 90 days. Inputs include the balance, age of the debt, type of original creditor, payment history, whether a working phone number is on file, and the outcome of earlier contact attempts. High-scoring accounts go to the call floor first, while low-scoring accounts get letters and digital reminders. A monotonic constraint, which most GBDT libraries support, guarantees that the score never rises as the debt ages when everything else is equal. That matches how experienced collectors understand the business and makes the model easier to defend to the creditors whose accounts the agency works.
What to watch for
- Too many or too deep trees will overfit, so use early stopping against a separate validation set.
- Accuracy depends more on tuning the learning rate, tree depth, and sampling settings than it does with random forests, so budget time for a structured search.
- Boosting will find and exploit any input recorded after the outcome it predicts, so audit features for information leakage.
- Scores are often miscalibrated, especially after classes are rebalanced, so calibrate them before treating them as probabilities.