Volume I · The Linear Story
Bending the Line Just Enough
A single straight line assumes the whole dataset shares one relationship between x and y. Look at almost any real scatter of points and that assumption is visibly false — the true curve bends one way on the left and another on the right. Locally weighted regression refuses the assumption outright: rather than fitting one line for the whole dataset, it fits a new line for every single point you ask it about.
The mental model
Keep the entire training set instead of compressing it into θ, and for every new query point, refit a fresh regression that only listens to the training examples nearby. The price of dropping the global-line assumption is that you must redo the fitting, from scratch, every single time you ask a question.
Parametric versus non-parametric
Lecture 1's linear regression is parametric: it fits a fixed-size parameter vector θ once, and from that point on the training data itself can be thrown away — every future prediction uses only θ. That is efficient, but it commits, once and for all, to the shape a single line can express.
Locally weighted regression (LWR) is non-parametric in the specific sense that the amount of "stuff" you need to keep around grows with the size of the training set, not with a fixed parameter count. The training set is never discarded. Instead, for every new query point x, LWR runs a fresh weighted regression, and that regression only trusts training examples close to x.
The weighted cost function
Ordinary linear regression minimises Σi(θTx(i) − y(i))² — every training example counts equally, regardless of where the query point is. LWR instead minimises a weighted version, where each training example carries its own weight w(i):
The weight is chosen to fall off with distance from the query point x, and the standard choice is a Gaussian bump centred on x:
A training example sitting almost on top of the query point gets w(i) ≈ 1 and is fit almost exactly; one far away gets w(i) ≈ 0 and is effectively ignored for this particular query, even though it still matters for a different query elsewhere. τ is the bandwidth — a smaller τ narrows the bump and makes the fit more local (and more jagged); a larger τ widens it, and as τ → ∞ every weight tends to 1 and LWR reduces exactly to ordinary linear regression.
The trade this makes explicit
This is the lecture's real teaching point, so it is worth stating directly rather than leaving it implicit. LWR makes almost no global assumption about the shape of the relationship between x and y — with a small enough τ, it can trace almost any curve. That freedom is not free: it costs keeping and re-weighting the entire training set for every single prediction, where parametric linear regression needed only θ. This is a genuine bias/computation trade, named directly here for the first time in this track, and it will recur under a sharper name.
Where this goes next
Volume IV formalises this exact trade as bias and variance. A model with almost no assumed structure — LWR with a small τ — can fit the training data nearly perfectly, which is low bias, but its fit is highly sensitive to exactly which training points happened to fall nearby, which is high variance. A model with a strong global assumption — Lecture 1's single line — sits at the opposite end: high bias, because it cannot bend to a curve no matter how much data it sees, but low variance, because which particular points it saw barely changes the fitted line. Lecture 9 gives this trade-off a number.
Note what LWR does not change: the cost function being minimised at each query point is still a weighted sum of squared errors, and the closed-form solution is still a normal-equations-style computation — a weighted version of Lecture 1's θ = (XTX)−1XTy, with a diagonal weight matrix inserted between the two X terms. Nothing about the fitting mechanics is new; what changes is that it happens once per query rather than once for the whole dataset, and the weights decide, fresh each time, which examples get a say.