Volume V · Regularizing and Choosing

Penalizing Complexity on Purpose

Two lectures established that a smaller hypothesis class generalises more predictably, and left the actual class fixed throughout. Nothing so far tells the optimiser to prefer a smaller weight over a larger one that fits the training data equally well. Regularization is the missing instruction, and it turns out to have a name in the language of prior beliefs, not just penalty terms.

L2 regularization as a prior, not a penalty

Every hypothesis class considered in this course so far has been fit by maximum likelihood: pick the θ that makes the observed data most probable, full stop. Ridge regression, the L2-regularized version of ordinary least squares, is what happens when a Bayesian prior belief is folded into that same estimation. Place a Gaussian prior directly on the parameters themselves:

and instead of the plain maximum likelihood estimate, take the maximum a posteriori (MAP) estimate — the θ that maximises the posterior probability of the parameters given the data, which by Bayes' rule is proportional to the likelihood of the data times this prior. A zero-mean Gaussian prior says, before any data is seen, that large values of θ are considered less likely than small ones. Maximising a likelihood weighted by that belief pulls the fitted θ toward zero relative to the plain MLE — exactly the shrinkage that appears, in the non-Bayesian telling, as an L2 penalty term added directly to the cost function.

The two descriptions are the same optimisation problem seen from different directions. "Add λ‖θ‖² to the cost" is the mechanical version. "Believe, a priori, that large weights are unlikely, and take the MAP estimate under that belief" is the same arithmetic with a stated reason behind it. Ridge regression is not an arbitrary penalty bolted onto least squares; it is what least squares becomes once a specific, nameable prior is taken seriously.

L1 regularization and sparsity

Lasso, introduced by Tibshirani, replaces the squared penalty with an absolute-value one — an L1 penalty on the parameters rather than L2. The standard interpretation, by analogy with the Gaussian story above, is that L1 regularization corresponds to placing a Laplace prior on the parameters rather than a Gaussian one, and taking the MAP estimate under that prior instead. That analogy is worth stating carefully as exactly that — the standard textbook interpretation — rather than a claim lifted verbatim from a specific source, since it was not confirmed word-for-word in the CS229 excerpt consulted for this lecture.

What is well established regardless of which prior story is preferred is the practical behaviour the two penalties produce. L1 regularization tends to drive many parameters to exactly zero, performing feature selection as a side effect of fitting — the resulting θ is sparse. L2 regularization shrinks every parameter smoothly toward zero without typically zeroing any of them out exactly; every feature keeps some influence, just a diminished one.

Cross-validation: the practical dial

Both penalties introduce a strength parameter — λ, or equivalently τ in the prior formulation — and nothing derived so far says what value it should take. Choosing it by looking at training error is self-defeating: training error keeps improving as regularization is relaxed, right up to the point where the bias-variance bargain from Lecture 9 starts working against you. Cross-validation is the practical answer, and it needs no held-out data beyond what you already have.

Split the training set into k disjoint folds. For each candidate regularization strength, train on k − 1 of the folds and evaluate on the one held out, then average that held-out performance across all k choices of which fold to hold out. The regularization strength with the best average held-out performance is the one to use for the final fit. The CS229 notes name k = 10 as a common default choice.

Tying the volumes together

Volume IV proved, twice over, that a bias-variance trade-off must exist and gave it a number for both finite and infinite hypothesis classes. This lecture supplies the practical machinery for navigating that trade on purpose. Regularization strength is the dial the theory says must exist somewhere between an under-fit, high-bias model and an over-fit, high-variance one; cross-validation is how that dial actually gets turned, using only the data already in hand, without ever peeking at genuinely new examples.