What is a random forest?
A random forest is an ensemble learning method that combines hundreds of decision trees into one model. A single deep tree fits its training data closely and changes a great deal when the data changes. A forest grows many such trees on different random views of the data and combines them, so their individual errors cancel out while the patterns they share remain. For classification the trees vote, and the share of votes serves as a score. For regression their predictions are averaged.
How a random forest works
Two sources of randomness keep the trees different from one another. First, each tree trains on a bootstrap sample, a random draw of rows with replacement that is the same size as the original data. Training on bootstrap samples and combining the results is called bagging, short for bootstrap aggregating. Second, at each split a tree considers only a random subset of the features, so one dominant variable cannot shape every tree. Each bootstrap sample leaves out roughly a third of the rows, and scoring those out-of-bag rows gives a built-in accuracy estimate. The main settings are the number of trees, the number of features tried at each split, and the minimum leaf size, and the defaults work well more often than with most methods.
Random forests in practice
A city housing department has a handful of inspectors and thousands of registered rental properties. It trains a random forest on past inspection results, using building age, prior violations, tenant complaints, utility shutoffs, ownership changes, and tax delinquency. Each week the forest ranks uninspected properties by the likelihood of serious violations, and supervisors assign inspectors from the top of the list. The department also inspects a random sample each month, because a model trained only on properties someone chose to inspect would learn the old targeting habits.
What to watch for
- Forest scores are often poorly calibrated, so check them before presenting them to users as probabilities.
- The default importance ranking is biased toward continuous variables and categories with many levels, so confirm the key drivers with permutation importance.
- Hundreds of deep trees make large model files and slower scoring, which matters for real-time inference.
- Tuned gradient-boosted trees often beat a forest on accuracy, so compare the two before going to production.