Volume IV · What Learning Theory Actually Guarantees

The Bias-Variance Bargain

Fit the same nine points twice — once with a straight line, once with a curve so flexible it visits every point exactly — and you get two models that agree perfectly on the data you showed them and disagree almost everywhere else. Only one of those disagreements is a surprise, and this lecture is about which one, and why it can be bounded before either model ever sees a tenth point.

Two ways to be wrong

Lecture 1 fit one straight line through every point in the training set, no matter how the data curved. Lecture 2 fit a separate local line for every query point, weighted so tightly by bandwidth τ that a small shift in which points happened to land in the sample could swing the fitted curve substantially. Those two lectures were the same trade-off stated twice, in opposite directions, and it is worth naming precisely now that both examples are on the table.

A plain linear fit has high bias: if the true relationship between x and y is not a line, the model is systematically wrong in the same way no matter how much data you feed it — more examples narrow down which line, but every line in that hypothesis class is the wrong shape. What it buys in exchange is low variance: draw a new training set from the same distribution and refit, and the line barely moves, because every point contributes to a single global average.

Locally weighted regression with a small τ sits at the opposite end. Bias is low — the fit can bend to follow almost any local shape the data actually has. Variance is high — because each prediction leans on only the handful of nearby points, and exactly which points fall nearby is a property of the particular sample you drew, a different sample redraws the curve.

Empirical risk minimisation

To make the trade precise, name the procedure every learning algorithm in this course actually runs. Given a hypothesis class H and a training set of m examples, empirical risk minimisation (ERM) picks the hypothesis ĥ ∈ H — equivalently the parameter θ — that minimises error measured on that training set directly. Write ε̂(h) for that quantity, the fraction of training examples h gets wrong, and call it the training error. Write ε(h) for the generalisation error: the probability that h is wrong on a fresh example drawn from the same distribution, one it never saw during fitting.

ERM's whole promise rests on ε̂ being a decent stand-in for ε — that doing well on the m examples in hand says something about doing well on the (infinite) rest of the distribution. That promise needs a proof, not an assumption, and the proof starts with a single fixed hypothesis before it ever touches ERM's choice among many.

Hoeffding's inequality

Fix one hypothesis h, chosen in advance, before looking at any data. Each training example either is or is not misclassified by h — an independent coin flip with some fixed probability of heads equal to ε(h). The training error ε̂(h) is just the average of m such flips. A concentration bound on averages of bounded independent random variables — Hoeffding's inequality — says this average cannot stray far from its true mean for long, and quantifies exactly how fast "far" becomes unlikely as m grows:

Read the right-hand side as a promise with a knob and a price. The knob is m: as the number of training examples grows, the bound shrinks — and it shrinks exponentially, not merely at some slow polynomial rate. The price is that this promise is only about the one hypothesis h that was fixed before any data was seen. It says nothing yet about a hypothesis chosen after looking at the training set — which is exactly what ERM does, every time.

Uniform convergence, over a finite class

Suppose the hypothesis class H has only k members, h₁ through h_k. Apply Hoeffding's inequality to each one separately, then take a union bound: the probability that at least one of the k hypotheses has a training error off by more than γ is at most k times the per-hypothesis probability. That single move — one Hoeffding bound per hypothesis, unioned over all k of them — is the entire derivation, and it delivers a bound that holds for every hypothesis in H at once, including whichever one ERM ends up picking:

Read this as ERM's actual guarantee. ĥ is whichever hypothesis ERM selected by minimising ε̂ on the training set. The bound says its true error is within a specific, computable margin of the best possible error achievable anywhere in H — and that margin grows with k, exactly because the union bound had to spread its probability budget δ across more hypotheses. A bigger hypothesis class gives ERM more room to find something that fits the training set unusually well by luck rather than by genuinely generalising, and the bound's extra slack is the price of that extra room.

Sample complexity

Turn the same inequality around and ask a different question: not "how tight is the bound at this m", but "how large must m be to guarantee a given tightness γ with confidence 1 − δ." Solving the uniform convergence bound for m gives:

The clean fact worth remembering is what appears inside the logarithm. The required sample size grows only logarithmically in k, the size of the hypothesis class. Doubling the number of hypotheses under consideration does not double the data you need — it adds a single, fixed increment. That is a genuinely forgiving rate, and it is the reason a hypothesis class can be allowed to grow fairly large before the sample-complexity cost becomes painful.