Volume VI · Trees and Committees

Splitting on What Matters Most

Every previous model in this course committed, up front, to a global shape — a line, a sigmoid, a Gaussian per class. A decision tree commits to nothing. It looks at whatever data is in front of it, asks the single most useful yes-or-no question it can find, and asks again on whichever half of the data that question left behind.

The mechanism

Formalised as CART — Classification and Regression Trees, following Breiman, Friedman, Olshen and Stone — the algorithm is a recursion with one greedy step repeated at every node. At the root, consider every feature and, for each feature, every possible threshold that could split the data into two groups. For each such candidate split, measure how much it reduces impurity across the two resulting child nodes compared with the impurity of the parent. Choose whichever single split reduces impurity the most, partition the data accordingly, and recurse on each child exactly the same way — considering every feature and every threshold again, from scratch, on whatever subset of the data now sits at that node.

Recursion stops at some stopping criterion: a maximum tree depth, a minimum number of examples remaining at a node, or a node whose examples are already pure enough that no split would help. Nothing about this procedure looks ahead — each split is chosen to be the best available right now, with no consideration of how it constrains splits two or three levels further down. That greediness is both the entire source of the algorithm's simplicity and, as the closing section argues, the entire source of its main weakness.

Measuring impurity

"Reduces impurity" needs a number to be greedy about. The CS229 notes give cross-entropy as the splitting criterion: for a region R containing examples from C possible classes, let p̂_c be the fraction of examples in R belonging to class c, and define:

A region containing only one class has p̂_c = 1 for that class and 0 for every other, and the sum evaluates to zero — perfectly pure, zero impurity. A region split evenly across classes has every p̂_c away from 0 and 1, and the sum is large — maximally impure. A candidate split is scored by how much this quantity drops, weighted by size, going from the parent region to its two children; the split chosen at each node is whichever one drops it the most.

Gini impurity is mentioned in the CS229 notes as a closely related alternative criterion. Its standard form — stated here as the standard form, since it was not verbatim-confirmed from the fetched source this session — is:

Both criteria are zero exactly when a region is pure and positive whenever classes are mixed together; which one a particular implementation uses rarely changes the resulting tree by much, since both are measuring the same underlying idea of "how mixed is this region" from slightly different angles.

The weakness this creates

A single tree grown deep enough to keep splitting until every leaf is pure will fit its training set almost exactly — impurity driven to zero everywhere. That is precisely the high-variance failure mode named in Lecture 9. Because each split is chosen greedily from whatever examples happen to be in the current region, a small change in the training data — a handful of examples added, removed, or reweighted — can alter an early split, and altering an early split changes which examples reach every node beneath it. The tree that results can look substantially different from the one grown on a near-identical sample, even though the underlying relationship being learned has not changed at all.