Volume III · Margins and Kernels

The Trick That Isn't Really a Trick

Rewrite yesterday's optimisation problem in a different but equivalent form, and something falls out that was never assumed: the optimal classifier only ever needs to know how similar two points are to each other, never what either point actually looks like on its own. Once you notice that, you can swap in a different notion of similarity entirely — and classify in a space you never explicitly build.

The tool: Lagrangian duality

Lecture 7 left the optimal margin classifier as a constrained minimisation: shrink ‖w‖ subject to every training point clearing a functional margin of 1. Constrained optimisation problems of this shape have a standard machinery for rewriting them — Lagrangian duality, together with the Karush-Kuhn-Tucker (KKT) conditions that characterise an optimal solution. The canonical reference for both is Boyd and Vandenberghe's Convex Optimization, and everything in this lecture leans on that machinery rather than inventing new tools.

The mechanics of Lagrangian duality are not this lecture's subject in their own right — they are the standard convex-optimisation apparatus for turning a constrained "primal" problem into an equivalent "dual" one, by introducing one Lagrange multiplier α_i per constraint and optimising over those multipliers instead of over w and b directly.

The dual problem

Apply that machinery to the optimal margin classifier from Lecture 7, and the primal minimisation over (w, b) turns into a maximisation over a set of multipliers α_i, one per training point.

Look carefully at what the training data contributes to this objective: every single term involving x appears only inside an inner product ⟨x⁽ⁱ⁾, x^(j)⟩ between two training points. The raw feature vectors never appear on their own, individually, anywhere in the dual objective — only pairwise inner products between them. That fact was not assumed going in; it is what the Lagrangian rewriting revealed.

Support vectors, via complementary slackness

The KKT conditions include a fact about the optimal multipliers called dual complementarity, which pins down exactly which training points end up mattering.

Here g_i(w) ≤ 0 is the way the i-th margin constraint is written for the KKT machinery, and the product being zero means one of two things has to hold for every point: either α_i* = 0, or the constraint g_i(w*) is exactly tight — meaning the point sits exactly on the margin boundary. So a nonzero α_i can only ever occur at a point lying precisely on the margin. Every other training point — every one that sits strictly inside its class's side of the street — is forced to α_i = 0 and contributes nothing at all to the final classifier. The points with nonzero α_i are exactly the support vectors — the training examples that give the algorithm its name.

The kernel trick

Because the entire dual objective — and, it turns out, the final prediction rule built from it — depends on the data only through inner products ⟨x, z⟩, there is nothing stopping you from replacing that inner product with a different function entirely, as long as that function still behaves like a valid inner product somewhere.

Here φ is some feature mapping — possibly into a very high-dimensional space, possibly into an infinite-dimensional one — and K computes the inner product as if both points had already been mapped through φ, without ever computing φ(x) or φ(z) explicitly. Every place ⟨x⁽ⁱ⁾, x^(j)⟩ appeared in the dual problem, substitute K(x⁽ⁱ⁾, x^(j)) instead, and the entire derivation from §8.2 continues to hold — the optimisation never needed anything more than the numbers K returns.

What licenses treating an arbitrary function K as a legitimate inner product in some feature space is Mercer's theorem: K is a valid kernel — meaning it does correspond to an inner product φ(x)^Tφ(z) for some mapping φ — if and only if its Gram matrix (the matrix of K evaluated on every pair from any finite sample of points) is symmetric and positive semi-definite. That single algebraic check is what lets you propose a similarity function and know, in advance, that some feature space exists in which it is literally an inner product — without ever having to construct that space by hand.

Making room for imperfect data: the soft margin

Every derivation so far assumed the two classes could be separated perfectly. Real data is rarely so obliging — a handful of points may sit on the wrong side of any plausible boundary. The soft-margin SVM handles this by allowing individual points to violate their margin, at a cost.

Each slack variable ξ_i measures how badly point i violates its margin — zero if it meets the margin cleanly, positive if it encroaches on the street or is outright misclassified. The constant C sets the exchange rate between the two things being traded off: a wider margin, which the (1/2)‖w‖² term still wants, against a smaller total amount of slack, which the C∑ξ_i term penalises. A large C punishes margin violations heavily and pushes toward the hard-margin case; a small C tolerates more violations in exchange for a wider street.

Solving it in practice: SMO

The dual problem's equality constraint — ∑α_i y⁽ⁱ⁾ = 0 — creates a genuine obstacle for a naive coordinate-ascent approach: you cannot update a single α_i in isolation while holding all the others fixed, because changing it alone would violate the constraint. Something else has to move to compensate.

Platt's 1998 Sequential Minimal Optimization (SMO) algorithm resolves this with the smallest working set that can possibly satisfy the constraint: it repeatedly selects a single pair (α_i, α_j) and updates that pair jointly — holding every other α fixed — in a way that keeps the equality constraint satisfied, because a change in α_i can always be offset by an equal and opposite adjustment to α_j. Each such pairwise update has a closed-form analytic solution, so no general-purpose quadratic-programming solver is needed at all — the whole optimisation reduces to repeating this cheap, exact two-variable update until convergence.

Closing the volume

Put the pieces of this lecture back together in order. The dual formulation was not chosen for elegance — it was the mechanical consequence of applying Lagrangian duality to Lecture 7's margin-maximisation problem. That consequence revealed, honestly, that the optimisation depends on the data only through inner products. Once that was true, substituting a kernel for the inner product was simply following the algebra where it already pointed. The soft margin and SMO exist for the same unglamorous reason every practical method exists: real data is not perfectly separable, and a general QP solver is not the cheapest way to fit millions of α's. None of it is magic. All of it was earned in §8.2.