Volume I · The Linear Story

From a Line to a Boundary

θ^Tx produces any real number, from far negative to far positive. A yes/no decision needs a probability between 0 and 1. The gap between those two facts is closed by a single function that squashes any real number into a valid probability — and once that function is in place, the calculus for finding the best θ barely changes at all.

The sigmoid

Define the logistic function, also called the sigmoid: g(z) = 1 / (1 + e^−z). As z → +∞, e^−z → 0 and g(z) → 1; as z → −∞, e^−z → ∞ and g(z) → 0; at z = 0, g(z) = 1/2 exactly. It is smooth, monotonic, and maps the whole real line onto the open interval (0, 1) — exactly the range a probability needs. The logistic regression hypothesis composes this with the familiar linear score:

h_θ(x) is read directly as a probability: P(y = 1 | x; θ).

The likelihood, derived

Assume y given x is Bernoulli-distributed with success probability h_θ(x): P(y = 1|x;θ) = h_θ(x) and P(y = 0|x;θ) = 1 − h_θ(x). These combine into a single expression valid for both cases at once — a standard trick for Bernoulli variables — by using y itself as an exponent:

Check it: set y = 1 and the second factor's exponent is 0, collapsing it to 1, leaving h_θ(x). Set y = 0 and the first factor's exponent is 0, leaving 1 − h_θ(x). Both cases recover the assumption exactly. With m independent training examples, the log-likelihood of the whole dataset — equivalently, up to sign, the cross-entropy objective — is:

The gradient — and a genuine surprise

Maximising ℓ(θ) by gradient ascent requires ∂ℓ/∂θ_j. The derivative of the sigmoid has a famously clean closed form, g′(z) = g(z)(1 − g(z)), and working through the chain rule for a single training example with that identity — differentiating log h(x) and log(1 − h(x)) separately, then combining — collapses, after the algebra, to exactly (y − h_θ(x))x_j. Summed over the training set and folded into a gradient-ascent step, this gives the update rule:

Set this beside Lecture 1's LMS rule and the two are identical in form, term for term. This is worth dwelling on rather than passing over: the hypothesis function itself changed completely, from the linear θ^Tx to the nonlinear sigmoid g(θ^Tx), and the cost function changed from squared error to a log-likelihood — and yet the update rule that maximum likelihood hands back is the same "error times input" shape in both cases. That is not a coincidence to be shrugged at; it is a consequence of the exponential-family structure both models share, made fully explicit in Lecture 4.

Newton's method — faster, at a cost

Gradient ascent takes a step in the direction of steepest local increase, but it only consults the first derivative — the local slope — and so it takes many small steps. Newton's method additionally consults the second derivative — the local curvature — and reaches the optimum in far fewer iterations for objectives like this one. In the multivariate case, the update replaces the scalar second derivative with the Hessian matrix H, where H_ij = ∂²ℓ(θ)/∂θ_i∂θ_j:

The reason this converges so much faster is precisely that it uses curvature, not just slope — a step that already knows the cost surface is flattening out nearby can slow down or turn appropriately, where gradient ascent only ever knows which way is locally uphill. The cost is computing and inverting the n×n Hessian at every single step, an O(n³) operation that becomes prohibitive at the large n gradient ascent tolerates comfortably — the same exactness-versus-scale trade seen already between the normal equations and LMS.

The perceptron: a caveat, not a special case

Swap the sigmoid for the hard threshold g(z) = 1{z ≥ 0} — one if z is non-negative, zero otherwise — and the identical-in-form update rule θ_j := θ_j + α(y⁽ⁱ⁾ − h_θ(x⁽ⁱ⁾))x_j⁽ⁱ⁾ still applies. This is the perceptron learning algorithm, and it predates logistic regression's use in statistics by decades. The caveat matters and should not be glossed over: because g(z) here is a discontinuous step rather than a smooth probability, there is, per the CS229 notes, no probabilistic or maximum-likelihood interpretation attached to this update rule at all. Its resemblance to the LMS and logistic-regression rules is a fact about the algebra of "error times input" updates in general, not a hidden case of the maximum- likelihood story this lecture otherwise tells. The perceptron is presented here for its historical importance, not as a third member of that family.