Volume I · The Linear Story

The Line That Learns Itself

Fitting a line to data is usually handed over as a formula to memorise — invert this matrix, multiply by that vector, done. It is actually two different ideas wearing the same coat, and a third idea hiding underneath both that explains why anyone thought squared error was the right thing to minimise in the first place.

The mental model

There is an iterative way to fit a line and a closed-form way, and they agree — because both are secretly maximising the likelihood of the data under a single assumption about noise. See the assumption, and squared error stops being a convention and becomes a consequence.

The hypothesis and the cost

Start with the object being fit. Given an input x with n features, the hypothesis is a linear function of those features, weighted by a parameter vector θ: h_θ(x) = θ^Tx = Σ_j θ_jx_j. The data supplying the fit is m training examples, written (x⁽ⁱ⁾, y⁽ⁱ⁾) for i = 1, …, m — the superscript here is an index into the training set, not an exponent, a convention worth fixing now because it recurs for the rest of this course. x⁽ⁱ⁾ is the i-th input, y⁽ⁱ⁾ its target.

"Fitting" means choosing θ to make h_θ(x⁽ⁱ⁾) close to y⁽ⁱ⁾ across all m examples, and "close" is made precise by a cost function — half the sum of squared errors:

The cost function J(θ) = (1/2) Σ_i=1^m ( h_θ(x⁽ⁱ⁾) − y⁽ⁱ⁾ )²

Everything in this lecture is one of two ways to make J(θ) small, plus a reason the squared term belongs there at all rather than, say, an absolute value or a fourth power.

LMS: descending the cost surface

The first way is to start θ somewhere — the zero vector will do — and repeatedly change it in the direction that decreases J(θ) fastest, which is the negative gradient. Working the derivative out for a single training example makes the update concrete. Differentiate (1/2)(h_θ(x) − y)² with respect to θ_j: the chain rule pulls out a factor of (h_θ(x) − y), and the inner derivative of θ^Tx with respect to θ_j is just x_j. So ∂J/∂θ_j = (h_θ(x) − y)x_j for one example, and stepping in the negative-gradient direction with a learning rate α gives the update rule:

LMS (Widrow-Hoff) update rule θ_j := θ_j + α ( y⁽ⁱ⁾ − h_θ(x⁽ⁱ⁾) ) x_j⁽ⁱ⁾

Read the sign as a diagnosis: when the hypothesis under-predicts, y⁽ⁱ⁾ − h_θ(x⁽ⁱ⁾) is positive and every θ_j nudges up in proportion to how much x_j⁽ⁱ⁾ contributed; over-predict and the nudge reverses. Apply this update to one example at a time and it is stochastic gradient descent; sum the gradient over all m examples before taking a step and it is batch gradient descent. Either way, the rule is called least mean squares, or the Widrow-Hoff rule after the engineers who used it for adaptive filtering decades before it was applied to learning problems by this name.

Figure 1a Three faint intermediate lines mark successive gradient steps, each rotating and shifting the fit to reduce J(θ); the solid line is the final position. Every one of these steps is the same local computation — how far off is the prediction at this point, and which way does nudging θ fix it.

The normal equations: skipping the walk

LMS reaches the minimum by taking many small steps. For linear regression specifically, that walk is optional: J(θ) is a convex quadratic function of θ — a bowl with a single lowest point and no other local minima to get stuck in — so the minimum can be located directly by setting its gradient to zero and solving. Collect the training inputs as rows of a design matrix X (m rows, one per example, n columns, one per feature) and the targets as a vector y. Solving ∇_θJ(θ) = 0 in this matrix form yields the normal equations:

The normal equations, solved θ = (X^TX)⁻¹ X^Ty

One matrix inversion and two multiplications, and θ is exact — no learning rate to tune, no convergence to wait for. The reason gradient descent remains the default in practice despite this closed form existing is cost: inverting an n×n matrix takes roughly O(n³) time, which is fine at n in the dozens and prohibitive once n reaches the thousands or millions that real feature sets routinely hit. LMS trades an exact answer for one that scales.

Two roads, one destination

Because J(θ) is convex, both routes provably reach the same θ — gradient descent approaches it iteratively, the normal equations land on it in one step. Neither is more "correct"; they are a trade between exactness and cost that recurs throughout this course, most explicitly in Lecture 3's contrast between gradient ascent and Newton's method.

Why squared error at all

Both derivations above simply accepted J(θ) as the thing to minimise. The deeper question is why squared error is the "right" cost rather than an arbitrary choice, and the answer is a probabilistic argument worth deriving in full, because it is the payoff the rest of this lecture has been building towards.

Assume the target is the linear function of x plus noise: y⁽ⁱ⁾ = θ^Tx⁽ⁱ⁾ + ε⁽ⁱ⁾, where the error terms ε⁽ⁱ⁾ are independent and identically distributed as ε⁽ⁱ⁾ ~ 𝒩(0, σ²) — zero-mean Gaussian noise, of the kind that arises from unmodelled effects and measurement error. This assumption fixes the conditional distribution of y given x:

The noise assumption, restated as a distribution y⁽ⁱ⁾ | x⁽ⁱ⁾; θ ~ 𝒩( θ^Tx⁽ⁱ⁾, σ² )

Treating θ as fixed but unknown, the probability of the observed y⁽ⁱ⁾ given x⁽ⁱ⁾ and θ is the Gaussian density evaluated at y⁽ⁱ⁾. Because the m training examples are independent, the likelihood of the whole dataset is their product:

The likelihood of θ L(θ) = Π_i=1^m p(y⁽ⁱ⁾ | x⁽ⁱ⁾; θ) = Π_i=1^m (1/√(2πσ²)) exp( − (y⁽ⁱ⁾ − θ^Tx⁽ⁱ⁾)² / 2σ² )

Maximum likelihood says: choose θ to make the observed data as probable as possible under the model. Products of exponentials are unpleasant to differentiate, so take the log — a monotonic transform that preserves the location of the maximum — turning the product into a sum:

The log-likelihood ℓ(θ) = log L(θ) = m·log(1/√(2πσ²)) − (1/σ²) · (1/2) Σ_i=1^m (y⁽ⁱ⁾ − θ^Tx⁽ⁱ⁾)²

Look at the second term. The first term and the 1/σ² factor do not depend on θ at all — they are constants as far as the maximisation is concerned. Maximising ℓ(θ) with respect to θ therefore reduces to minimising the one piece that does depend on θ: (1/2) Σ_i (y⁽ⁱ⁾ − θ^Tx⁽ⁱ⁾)². That expression is J(θ), unchanged, term for term.

Figure 1b A bell curve sketched at three points along the fitted line, each centred on the line itself. The model does not claim the data lies exactly on the line — it claims the true y at each x is a draw from a Gaussian whose mean is the line's value there. Squared error is what falls out of maximising the likelihood of that assumption.

That identity is the whole point: minimising J(θ) and maximising the log-likelihood under independent Gaussian noise are the same optimisation problem, algebraically, term by term. Squared error was never an arbitrary convenience — it is the exact loss that maximum likelihood hands you once you assume Gaussian noise, and no other loss is the maximum- likelihood answer under that assumption.

The throughline

This is the shape of most of what follows. Assume a probability model for how the data was generated, and maximum likelihood tells you exactly which loss to minimise — this lecture's route, and Volume II's route when the model is generative rather than discriminative. The alternative, taken in Volume III, is to state an optimisation problem directly and justify it on other grounds — a margin, a generalisation bound — without first naming a probability model at all. Two moves, and by the end of Volume III both will have been derived in full.

Read the primary source

CS229 Lecture Notes — Part I: Linear Regression. The LMS derivation, the normal equations, and the probabilistic interpretation, in that order.
CS229 Main Notes (consolidated). The full course notes this lecture series follows.
aman.ai — CS229 Linear Regression. A page-by-page transcription, useful for cross-checking the algebra.

❦