Volume VIII · Learning Without Labels

Throwing Away the Right Dimensions

Take a cloud of points scattered across a hundred measured features and most of what actually varies between them lives along a handful of directions — often fewer than ten. The rest of the hundred dimensions are, for most practical purposes, noise dressed up as precision.

The mental model

Principal component analysis asks which directions in a dataset carry the most spread, and answers with the eigenvectors of a single matrix. The striking part is that a completely different question — which directions lose the least information when you throw the rest away — asks for exactly the same eigenvectors, by way of an entirely separate derivation.

The question, made precise

Start with m examples x⁽¹⁾, …, x^(m) in n dimensions, centred so that their mean is zero. The question this lecture asks is deceptively simple: is there a single direction — a unit vector u, with ‖u‖ = 1 — such that projecting every point onto it preserves as much of the data's spread as possible? Project x⁽ⁱ⁾ onto u and you get a single number, x^(i)ᵀu, the coordinate of that point along the line through the origin in direction u. The variance of those projected coordinates, across all m points, is what "spread preserved" means here.

Write that variance out and it becomes an optimisation problem over u:

Variance of the projection maximise (1/m) Σ_i ( x^(i)ᵀu )² = uᵀ Σ u, subject to ‖u‖ = 1

where Σ = (1/m) Σ_i x⁽ⁱ⁾(x⁽ⁱ⁾)ᵀ is the empirical covariance matrix of the data — an n × n matrix built entirely from the dataset, with no labels anywhere in it. This is the first algorithm in the course that never once mentions y.

Figure 17a The cloud is stretched noticeably more along one diagonal than any other direction. That direction is the first principal component — the unit vector u that maximises the variance of the projected points.

Solving it — Lagrange, once

The constraint ‖u‖ = 1 rules out simply letting u grow without bound to inflate uᵀΣu arbitrarily. Introduce a Lagrange multiplier λ for that constraint and form the Lagrangian:

The Lagrangian L(u, λ) = uᵀ Σ u − λ ( uᵀu − 1 )

Differentiate with respect to u and set the result to zero:

Stationarity condition ∇_u L = 2Σu − 2λu = 0 ⟹ Σu = λu

That single line is the entire derivation. Σu = λu is precisely the definition of an eigenvector: u is an eigenvector of the covariance matrix Σ, with λ its corresponding eigenvalue. Substitute this back into the objective — uᵀΣu = uᵀ(λu) = λ(uᵀu) = λ, since ‖u‖ = 1 — and the maximal variance achievable along any single direction is exactly λ, the eigenvalue itself. To maximise variance, pick the eigenvector with the largest eigenvalue.

The generalisation to k dimensions follows the same logic without a new derivation: to capture as much variance as possible using k orthogonal directions at once, take the top-k eigenvectors of Σ, ranked by eigenvalue from largest to smallest. Projecting the data onto the subspace they span is, among all k-dimensional subspaces, the one that keeps the most variance.

The other route — reconstruction error

Ask a seemingly different question of the same data. Rather than "which direction keeps the most spread", ask: "if I must compress every point down to a k-dimensional summary and then reconstruct an approximation of the original point from that summary, which k-dimensional subspace loses the least?" This is the standard equivalent formulation of PCA, and it is worth stating honestly rather than glossing over: minimise the total squared reconstruction error,

Reconstruction error, the standard equivalent formulation minimise Σ_i ‖ x⁽ⁱ⁾ − x̂⁽ⁱ⁾ ‖²

over every possible rank-k reconstruction x̂⁽ⁱ⁾ built from an orthonormal basis of k directions. Solved out, this objective lands on exactly the same top-k eigenvectors of Σ as the variance-maximisation route above.

Figure 17b Every point dropped onto the same direction u found in Figure 17a. The perpendicular distance from each point to the line is exactly what the reconstruction-error objective is minimising — and projecting onto this particular direction loses less total distance than projecting onto any other line through the origin would.

The point worth sitting with is not that these two objectives happen to be compatible. It is that they are, on their face, aiming at completely different things — one wants the projected data to stay as spread out as possible, the other wants the discarded detail to be as small as possible — and yet solving either one exactly produces the identical answer: the top-k eigenvectors of the same covariance matrix Σ. That equivalence is a small, satisfying proof in its own right, not a coincidence to wave past on the way to the next lecture.

The question PCA raises but does not answer

PCA finds directions that are uncorrelated — the eigenvectors of Σ are, by construction, orthogonal to one another. Uncorrelated is a weaker condition than independent. A natural next question is what happens if the goal is maximally independent components rather than merely uncorrelated ones — the territory of Independent Component Analysis. A single canonical citation for ICA was not confirmed this session, so it is named here only as the question PCA naturally raises, not as a result with a specific paper attached.

Where this sits in the course

This track's own landing page names three costs every algorithm in the course pays against: the generalisation problem, the optimisation problem, and the labels problem — what to do when supervision runs out. PCA is this lecture's answer to the third. There is no y anywhere in either derivation above; the covariance matrix Σ is built entirely from the x's. When nothing in the data tells you which dimensions matter, variance — or, equivalently, reconstruction fidelity — is the criterion this lecture uses to decide for you.

Read the primary source

CS229 Lecture Notes — Principal Component Analysis — Stanford. The variance-maximisation derivation and the Lagrange-multiplier argument in full.
CS229 · PCA — Aman.ai. Cross-verified transcription of the same derivation, useful as a second pass.

❦