Volume I · The Linear Story

One Framework, Many Regressions

Linear regression and logistic regression look like two unrelated algorithms that happen, suspiciously, to share an update rule. Lecture 3 flagged the resemblance without explaining it. It is not a coincidence — both are the same template, filled in with two different probability distributions, and the template itself hands you a third algorithm the moment you fill it in with a distribution over more than two classes.

The exponential family

A distribution belongs to the exponential family if its density can be written in this form, for some functions b, T and a:

η is the natural parameter, T(y) the sufficient statistic (in the cases used here, simply T(y) = y), and a(η) the log partition function, which normalises the density so it integrates to 1. b(y) is a base measure that does not depend on η. Many of the distributions used throughout this course — Gaussian, Bernoulli, Multinomial, Poisson — can be written in this exact shape, each with its own b, T and a.

A Generalized Linear Model (GLM) is built from an exponential-family distribution by three design choices, stated together because each is needed for what follows:

Linear regression, recovered

Choose the Gaussian as the exponential-family member, with variance fixed (a detail that can be set aside here, since it does not affect the mean being predicted). Writing the Gaussian density in exponential-family form and reading off its natural parameter gives the simplest possible relationship: η = μ, the mean itself. The link between the natural parameter and the mean is thus the identity function. Combined with GLM design choice 3, η = θ^Tx, and choice 2, predict E[y|x] = μ = η, this gives:

That is exactly Lecture 1's hypothesis, arrived at without ever assuming a linear shape by hand — it fell out of choosing Gaussian and applying the template.

Logistic regression, recovered

Choose the Bernoulli distribution, with mean φ = P(y=1). Writing the Bernoulli density in exponential-family form and solving for its natural parameter gives the log-odds: η = log(φ / (1 − φ)). Inverting this relationship for φ in terms of η gives:

Substitute η = θ^Tx from design choice 3 and this is, term for term, the sigmoid from Lecture 3 — h_θ(x) = 1/(1+e^−θᵀx). The sigmoid was not an arbitrary squashing function chosen because it happened to map onto (0,1); it is the specific link function the exponential family hands you once you commit to the Bernoulli distribution as the model for y.

Softmax regression, for free

The payoff for building the template rather than memorising two special cases: choose the Multinomial distribution — a draw among k classes rather than two — as the exponential family member, apply the identical three design choices, and the GLM this produces is softmax regression:

Nothing new had to be invented to reach this. Once Gaussian → identity link and Bernoulli → sigmoid link are understood as instances of one procedure, Multinomial → softmax is the same procedure run a third time, with a different distribution supplying different b, T and a. The multiclass case was not designed separately; it was waiting inside the template the whole time.

The payoff, stated plainly

This is not three separate algorithms that happen to resemble one another, and it is not three special cases awkwardly unified after the fact for tidiness. It is one template — exponential family plus three design choices — instantiated three times, and the "right" hypothesis function in each instantiation is a derivation, not a design decision made by hand. Lecture 3's surprise, that a nonlinear sigmoid hypothesis produces the identical update-rule shape as a linear one, is resolved here: both are GLMs built from the same template, so of course the mechanics rhyme.