Volume VIII · Learning Without Labels

Grouping Without Being Told the Groups

Every algorithm in this track so far was handed a label for every training example. This one is handed nothing but a scatter of raw points, and has to invent the groups itself — by guessing where the centres are, checking who is closest to each guess, and correcting the guess until it stops moving.

The mental model

K-means alternates two simple steps — assign each point to its nearest current centre, then move each centre to the mean of the points assigned to it — and a distortion function that only ever decreases is the reason this simple back-and-forth is guaranteed to settle down, if not necessarily onto the best possible answer.

No labels to fit against

Linear regression fits a line against y⁽ⁱ⁾. Logistic regression fits a boundary against y⁽ⁱ⁾. Every method through Volume VII of this track had a target to check its output against. Clustering removes that target entirely: the input is a set of points x⁽¹⁾, …, x^(m), with no label attached to any of them, and the task is to partition the points into k groups such that points inside a group resemble each other more than points in different groups do. Nobody tells the algorithm what the groups are, or even confirms afterward whether it found the "right" ones — there is no y⁽ⁱ⁾ to compare against.

The two alternating steps

K-means proposes k centres, μ₁, …, μ_k, and improves them by alternating two steps until nothing changes. The first step hardens the current guess into an assignment; the second step uses that assignment to relocate the guess.

Assignment step c⁽ⁱ⁾ := argmin_j ‖ x⁽ⁱ⁾ − μ_j ‖²

Each point is assigned to whichever of the k current centres it is closest to, measured by ordinary squared Euclidean distance. Then, holding that assignment fixed:

Update step μ_j := ( Σ_i 1{c⁽ⁱ⁾=j} x⁽ⁱ⁾ ) / ( Σ_i 1{c⁽ⁱ⁾=j} )

Each centre moves to the mean of exactly the points currently assigned to it — the centre's new position is the plain average of its own cluster, nothing more elaborate. Repeating the two steps in sequence — assign, then relocate, then assign again — is the entire algorithm.

Figure 15a Left, the raw input: no colour, no label, no hint of a group. Right, after one assignment step: each point recoloured by which of three placed centres it is nearest to. K-means invents this colouring; nothing in the input specified it.

A quantity that can only shrink

The reason repeating these two steps eventually stops is that both of them are moves in a single optimisation, minimising one distortion function:

Distortion function J(c, μ) = Σ_i ‖ x⁽ⁱ⁾ − μ_c⁽ⁱ⁾ ‖²

Holding the centres μ fixed, the assignment step reassigns every point to whichever centre it is closest to — which, for each point individually, can only lower or leave unchanged that point's contribution to J, since any other assignment would use a farther centre. Holding the assignment c fixed, the update step moves each centre to the mean of its assigned points — the mean is precisely the point that minimises total squared distance to a fixed set of points, so this step, too, can only lower or leave unchanged J. Two steps that each never increase a quantity bounded below means the quantity converges: J decreases monotonically until it stops changing, at which point the algorithm has reached a fixed point of both steps at once.

That guarantee is weaker than it sounds. Convergence is to some fixed point of the two alternating steps — a local optimum of J — not necessarily the global one, and which local optimum k-means lands on depends on where the centres were initialised. In practice this is why k-means is run several times from different random initialisations and the run with the lowest final J is kept, rather than trusted on a single run.

Figure 15b Centres tighten around the true cluster structure round by round while the distortion function J falls monotonically — the convergence guarantee of §15.3 made visible.

A name, honestly qualified

This alternating assign-then-relocate procedure is widely known as Lloyd's algorithm. This lecture names it for that reason, but flags plainly that the specific historical citation for Lloyd's original paper was not independently re-verified in the preparation of this material — so the algorithm is named here without over-asserting a precise publication date or venue as settled fact. The mechanism in §15.2 and the convergence argument in §15.3 stand on their own regardless of that attribution's exact details.

What k-means assumes

This is the first fully unsupervised algorithm in the track, and it earns that position by making two assumptions worth naming rather than absorbing silently. First, it measures every point's fit to a cluster by plain Euclidean distance from a single centre, which implicitly assumes clusters are round-ish blobs rather than elongated or oddly shaped regions. Second, it requires k — the number of clusters — to be fixed in advance, rather than discovered from the data. Both assumptions are real restrictions, not incidental details, and both are relaxed by the probabilistic version of this idea taken up next: a model that replaces "nearest centre, hard assignment" with a proper probability of belonging to each cluster.

Read the primary source

Stanford CS229, Lecture notes on k-means clustering. cs229.stanford.edu/main_notes.pdf
Cross-verified transcription, CS229 — K-Means Clustering. aman.ai/cs229/kmc

❦