Volume IV · What Learning Theory Actually Guarantees

Counting What a Model Can Fit

Ask two people how expressive a model is and one will count its parameters. Count three points on a page instead, and try to draw every possible way of colouring them red and blue with a single straight line. That second count is a real definition — the first was never anything more than a proxy.

The problem the last lecture left open

Lecture 9 proved a clean sample-complexity result, and then admitted, at the very end, that it only applies to a finite hypothesis class of size k. Linear classifiers do not qualify. A linear classifier in ℝⁿ is parameterised by a real-valued θ, and there are uncountably many possible values of θ — the hypothesis class is infinite, k does not exist as a finite number, and log(2k/δ) is simply not defined. The bound that took a whole lecture to derive cannot be applied to almost any model this course actually cares about, as stated.

The fix is not to abandon the argument's shape — bound one hypothesis, union over the class, solve for m — but to replace k with a different measure of size, one that stays finite even when the literal count of hypotheses does not.

Shattering, defined precisely

That replacement is the VC dimension, named for Vladimir Vapnik and Alexey Chervonenkis. Given a hypothesis class H and a set of d points, say H shatters that set if, for every one of the 2^d possible ways of labelling those d points as positive or negative, some hypothesis in H achieves exactly that labelling. The VC dimension of H, written VC(H), is the size of the largest set of points that H can shatter.

Notice what this definition does and does not count. It never mentions the number of parameters in θ. It counts labellings — the number of genuinely distinct behaviours the class can be made to exhibit on a worst-case, favourably chosen set of points. Two hypothesis classes with wildly different parameter counts can have the same VC dimension; a class with infinitely many parameters can, in principle, have a small VC dimension, if most of that freedom does not translate into new labellings.

The worked example: linear classifiers in ℝ²

The canonical illustration, and the one worth carrying as the mental anchor for this entire idea, is linear classifiers in the plane. Take three points arranged in general position — not all on one line. There are 2³ = 8 ways to label three points red or blue. Every single one of those eight labellings can be achieved by some straight line: three-against-none, two-against-one in either of the three possible pairings, and the reverse of each. Three points, arranged well, are shattered.

Four points break the pattern. There exist configurations of four points — an XOR-like arrangement, where two diagonally opposite points are labelled positive and the other two negative — that no single straight line can separate. No amount of choosing the line differently rescues that one labelling, because a line's positive side is always a single connected half-plane, and the XOR pattern requires the positive class to occupy two disconnected corners at once. Since at least one labelling of four points is unreachable, linear classifiers in ℝ² cannot shatter any set of four points, and:

Sample complexity, extended to the infinite case

VC dimension is what makes the previous lecture's machinery reusable outside finite classes. In place of a bound that scaled with log(k) — meaningless once k is infinite — the analogous result for a hypothesis class with VC dimension d says that the number of training examples needed for uniform convergence scales as:

Sample complexity is governed by VC dimension, not by the raw number of parameters. A hypothesis class can have infinitely many real-valued parameters and still have a small, finite VC dimension — linear classifiers in ℝ² being exactly that case, with one continuous θ vector and only three points' worth of expressive freedom. The reverse is possible in principle too: nothing forces parameter count and VC dimension to move together, which is precisely why counting parameters was always the wrong proxy.