← Contents Volume II · Lecture 5

Volume II · Generative Learning

5

Learning the Classes, Not the Boundary

Every classifier so far in this series has asked one question: given a point, which side of a line is it on? Gaussian Discriminant Analysis asks a different question first — what does each class typically look like — and only afterwards, almost as an afterthought, draws the line.

The mental model

Logistic regression learns the boundary between classes directly. Gaussian Discriminant Analysis learns what each class typically looks like, and derives the boundary as a consequence — model the causes, not the decision.

Two ways to answer the same question

Logistic regression, the subject of Lecture 3, is a discriminative algorithm. It learns P(y|x) directly — or, in some discriminative algorithms, it skips probability altogether and learns a bare mapping from inputs straight to labels. Either way, the class boundary is the object being fit. Nothing in the model ever represents what the data belonging to a class actually looks like; it only ever tries to get the dividing line right.

A generative algorithm inverts the order of operations. Instead of modelling P(y|x) directly, it models P(x|y) — how each class generates the data you see — together with the class prior P(y). A prediction, when one is needed, falls out of Bayes' rule afterwards. The boundary is never fit; it is computed from two densities that were fit for an entirely different reason.

DISCRIMINATIVE data boundary, fit directly GENERATIVE P(x|y=0) P(x|y=1) boundary, derived after
Figure 5b The whole distinction in one picture. The left panel never draws a density — it draws a line and calls it done. The right panel draws two densities first; the line is simply where they happen to cross.

The GDA model, written down precisely

Gaussian Discriminant Analysis assumes a specific generative story for binary labels. The class itself is a coin flip, and each class's inputs are drawn from a multivariate Gaussian with its own mean but a covariance shared across both classes.

Class prior P(y) = φy (1 − φ)1−y
Class-conditional density (shared Σ) P(x|y=k) = 1 / ( (2π)n/2 |Σ|1/2 ) · exp( −(1/2) (x − μk)T Σ−1 (x − μk) )

Read the second line slowly, because every piece of it matters. It is a multivariate Gaussian in the n-dimensional input space, centred at μ0 for class 0 and at μ1 for class 1 — two different means, one for each class's typical location. But there is only ever one Σ. Both classes are assumed to have the same shape and orientation of spread; they differ only in where they sit.

TWO CLASS-CONDITIONAL DENSITIES P(x|y=0), centred at μ₀ P(x|y=1), centred at μ₁ boundary: where the densities cross
Figure 5a The boundary is not fit; it falls out where the two bell curves are equally likely. Move either mean and the boundary shifts on its own — nobody ever touches the line directly.

The consequence that makes GDA comparable to logistic regression

Apply Bayes' rule to the model above — combine the shared-covariance Gaussians with the class prior and simplify — and a remarkable fact appears: P(y=1|x) comes out to be exactly a sigmoid of a linear function of x. That is, under GDA's assumptions, the posterior has precisely the functional form that Lecture 3 posited outright for logistic regression.

What the GDA posterior reduces to P(y=1 | x; φ, μ0, μ1, Σ) = 1 / (1 + exp(−θTx))

Logistic regression assumed this sigmoid form as its starting point and fit θ to match it as directly as possible. GDA never assumes the sigmoid at all — it assumes something stronger, a full generative story about how the two classes' inputs are distributed — and the sigmoid falls out as a downstream consequence. This is the precise sense in which GDA makes the stronger assumption: logistic regression's model is implied by GDA's, not the other way round.

What the stronger assumption buys, and what it costs

A stronger assumption, if it happens to be true, is not free lunch turned to poison — it is genuinely valuable. When the data really is (approximately) Gaussian within each class with a shared covariance, GDA is asymptotically more efficient than logistic regression: it needs less training data to reach the same accuracy, because it is extracting more structure from every example it sees rather than treating the boundary as the only fact worth learning.

Logistic regression pays for its weaker assumption with a corresponding robustness. It never claims anything about the shape of P(x|y) — it only ever tries to get the boundary right directly — so it has nothing to be wrong about on that front. If a feature is actually Poisson-distributed within each class rather than Gaussian, GDA's core assumption is simply false, and logistic regression will typically win, because it never bet on the Gaussian shape in the first place.

That is the whole comparison, and it recurs throughout this volume: a generative model that assumes more about the world can be more data-efficient exactly when the assumption holds, and can lose to a discriminative model that assumed nothing exactly when it doesn't. Lecture 6 keeps the generative framing but swaps in a completely different assumption about P(x|y) — one that dispenses with Gaussians altogether.

Read the primary source

❦