Volume VIII · Learning Without Labels

The Algorithm That Guesses Twice

K-means makes a hard, all-or-nothing guess about which cluster each point belongs to and moves on. This lecture's algorithm makes a softer guess — a full probability of belonging to each cluster — and a single inequality from convex analysis guarantees that alternating between guessing and refitting can only ever improve the model's fit, never worsen it.

A hidden variable nobody observes

A Gaussian Mixture Model assumes each point x⁽ⁱ⁾ was generated in two stages: first, an unobserved choice of which of several Gaussian components generated it — a latent variable z⁽ⁱ⁾ — and second, a draw from that chosen component. Only x⁽ⁱ⁾ is ever seen; z⁽ⁱ⁾ is never observed directly, at training time or afterward. This is a harder setting than Lecture 15's: k-means never modelled an explicit probability of component membership at all, it simply hardened distance into a single nearest-centre assignment. Here, the very quantity we most want — which component generated this point — is precisely the part of the model we cannot see.

Jensen's inequality

The tool that makes the rest of this derivation work is a fact about convex and concave functions, independent of anything about mixtures or clustering:

The log function is concave, so the second form applies directly: the expectation of a log is at most the log of an expectation. Equality holds exactly when the random variable inside the expectation is (almost surely) constant — a fact the derivation below relies on directly, not just in passing.

The E-step

For each training example, the E-step computes the current model's best guess about the hidden component — not a hard choice, but the full posterior distribution over which component generated it, given the data and the current parameters θ:

This is the "soft" counterpart to k-means's assignment step: instead of one number c⁽ⁱ⁾ naming a single nearest centre, Q_i is an entire distribution — for a two-component mixture, something like "70% component 1, 30% component 2" for this particular point, rather than a forced choice of one or the other.

The M-step

The M-step refits the model's parameters, but not by maximising the true log-likelihood directly — that quantity is hard to optimise precisely because it involves summing over the unobserved z⁽ⁱ⁾ inside a logarithm. Instead, it maximises a specific lower bound on the log-likelihood, built directly from the E-step's guessed distributions Q_i:

Read the inner sum as an expectation under Q_i of a log-ratio — which is exactly where Jensen's inequality from §16.2 enters. Starting from the true log-likelihood log Σ_z⁽ⁱ⁾ P(x⁽ⁱ⁾, z⁽ⁱ⁾; θ), multiply and divide inside the sum by Q_i(z⁽ⁱ⁾), and recognise the result as log E_Qi[ P(x⁽ⁱ⁾, z⁽ⁱ⁾; θ) / Q_i(z⁽ⁱ⁾) ]. Applying the concave form of Jensen's inequality moves the log inside the expectation, giving E_Qi[ log( P(x⁽ⁱ⁾, z⁽ⁱ⁾; θ) / Q_i(z⁽ⁱ⁾) ) ] — precisely the M-step's objective — and Jensen's inequality guarantees this quantity is always less than or equal to the true log-likelihood it was derived from.

Why alternating the two steps can only help

The construction in §16.4 is only useful if the bound is tight exactly where it needs to be, and it is. Choosing Q_i to be the true posterior P(z⁽ⁱ⁾ | x⁽ⁱ⁾; θ) — precisely what the E-step computes — makes the ratio P(x⁽ⁱ⁾, z⁽ⁱ⁾; θ) / Q_i(z⁽ⁱ⁾) constant in z⁽ⁱ⁾, which is exactly the equality condition for Jensen's inequality named in §16.2. So immediately after an E-step, the M-step's lower bound touches the true log-likelihood exactly, at the current θ. The M-step then moves to whatever new θ maximises that lower bound. Because the bound touched the true curve at the starting point and the M-step can only move to a value of the bound at least as high as where it started, the true log-likelihood at the new θ is guaranteed to be at least as high as it was before — a genuine improvement in the bound is also a genuine improvement, or at worst no change, in the quantity we actually care about.

This is the same "guaranteed non-decrease" shape as Lecture 15's distortion function J, which could also only fall or hold steady across k-means's two alternating steps — but it is derived here from an entirely different piece of mathematics. K-means's guarantee came from a simple squared-distance argument: the mean minimises squared distance to a fixed set of points. EM's guarantee comes from Jensen's inequality applied to a concave log, and the fact that the E-step's choice of Q_i makes that inequality tight at exactly the point the M-step starts from.

Closing the comparison

Name the lecture's title precisely against Lecture 15's. K-means guesses once per point, hard, and moves on to relocating centres. EM guesses a whole probability distribution over possibilities every round — the "twice" refers to the two alternating steps themselves, guess then refit, not to guessing the same quantity a second time. The result is a strictly more general way of handling uncertainty about a hidden cluster assignment than k-means's hard nearest-centre rule, purchased at the cost of the fuller Jensen's-inequality argument this lecture needed to justify why it still provably improves, round after round.