Volume VI · Trees and Committees

Why a Crowd Beats an Expert

A single decision tree, grown deep enough to fit its training set exactly, has learned the data by heart rather than understood it. The fix is not a better tree. It is more trees — combined in one of two entirely different ways.

One tree, too much memory

Lecture 12 built a decision tree by asking one greedy question after another, each one chosen to reduce impurity the most. Grown deep, such a tree can carve out a leaf for every training example — zero training error, and a model that has essentially memorised the sample rather than learned the pattern inside it. That is the textbook definition of high variance: retrain the same tree on a slightly different sample of the same population and you get a noticeably different tree, because a single tree has no mechanism for ignoring the noise specific to the sample it happened to see.

Both ideas in this lecture accept that a deep tree is a high-variance estimator and do nothing to make any individual tree less greedy. Instead they change what happens after the tree is grown — one by averaging over many such trees, the other by training many such trees in a deliberate sequence.

Bagging: many guesses, averaged

Bagging — bootstrap aggregating — is mechanically simple. From a training set of m examples, draw a new sample of size m with replacement: a bootstrap resample, which on average contains about two-thirds of the original examples, some of them more than once. Train a full, deep tree on that resample. Repeat B times, on B independently drawn resamples, and average the resulting predictions (a plain mean for regression, a majority vote for classification).

The reason this helps is a fact about variance, not a fact about trees specifically. Each individual f_b is still exactly the high-variance estimator Lecture 12 warned about — bagging does nothing to tame any single tree. But the B trees were grown on different resamples, so their errors are only partially correlated with one another: where one tree overfits to noise present in its particular resample, another tree, trained on a different resample, is unlikely to overfit to the same noise in the same way. Averaging over many estimators whose errors are only partially correlated reduces the variance of the average below the variance of any one of them, without introducing new bias — the average of many unbiased-but-noisy guesses is still unbiased, and noticeably less noisy.

Random forests: decorrelating the crowd

Random Forests add exactly one further piece of randomness on top of bagging's bootstrap resampling: at each split, in each tree, the algorithm is only allowed to consider a random subset of the available features, rather than choosing freely among all of them. The split still picks the best available question from that restricted subset, by the same impurity-reduction rule as before — the restriction is on which features are even candidates, not on how the winner is chosen.

The reason this second layer of randomness earns its own name and its own paper is a correlation argument bagging's own reasoning did not anticipate. If one feature in the data is dramatically more predictive than the rest, then almost every bootstrap resample — no matter how it was drawn — will still support splitting on that same feature near the root, because the feature's predictive strength swamps the sampling noise. The bagged trees end up more correlated with one another than the variance argument in §13.2 assumed, which quietly erodes the benefit of averaging: correlated errors do not cancel the way independent ones do. Forcing each split to draw from only a random subset of features occasionally hides the dominant feature from a given split entirely, forcing that tree to discover a different, still-useful structure in the data. The trees decorrelate further, and the averaging step in §13.2 gets closer to its full theoretical benefit.

Boosting: a chain of specialists

AdaBoost abandons the averaging idea entirely and builds its ensemble as a sequence. Every training example starts with equal weight, w_i = 1/N. Each round fits a weak learner — often a tree with a single split, a "stump" — to the currently weighted training set, then measures how badly that weak learner did, weights its vote by how well it did, and reweights the training examples so the next round is forced to pay more attention to exactly what this round got wrong.

Read the reweighting rule literally: the exponent is zero for any example G_m classified correctly, so its weight is untouched, and it is α_m for any example G_m got wrong, so that example's weight is multiplied up by e^α_m — misclassified examples get heavier, and a weak learner that was already fairly accurate (small err_m, large α_m) upweights its mistakes especially hard. After M rounds, the final classifier is a weighted vote of every weak learner trained along the way:

This reweighting scheme is not an ad hoc heuristic. It can be shown to be exactly equivalent to forward stagewise additive modelling under the exponential loss L(y, ŷ) = e^−yŷ — at each round, AdaBoost is greedily adding the one weak learner and coefficient that most reduces the exponential loss on the current ensemble, holding everything fitted in previous rounds fixed. The weight update in the box above is what that greedy exponential-loss minimisation works out to in closed form.

From reweighting to gradients

Gradient boosting keeps AdaBoost's sequential structure — fit one weak learner, add it to the ensemble, repeat — but replaces the reweight-the-misclassified-examples mechanism with something more general. Instead of upweighting examples a previous round got wrong, each new weak learner is fit directly to the residuals of the current ensemble — or, more precisely, to the negative gradient of the loss function with respect to the ensemble's current predictions. Where AdaBoost's derivation is tied to one specific loss (exponential loss, as noted in §13.4), gradient boosting's functional-gradient-descent view works for any differentiable loss: squared error, absolute error, log-loss for classification, and beyond. AdaBoost is, in this sense, the special case of gradient boosting that falls out when the loss is fixed to be exponential.

Two crowds

The lecture's title is meant literally. Bagging and Random Forests build a crowd in the ordinary sense — many independently trained, roughly equal voters, whose collective judgment is simply their average. Boosting builds a different kind of crowd: a sequence of specialists, each one recruited for the specific job of covering the blind spots the previous specialists left behind. Both attack the weakness Lecture 12 diagnosed in a single deep tree, and both do so without asking any individual tree to be less greedy — they change what surrounds the tree, not the tree itself.