Volume III · Margins and Kernels

The Widest Street Between Two Classes

Draw two clusters of points on a page and hand someone a ruler. They can separate the clusters with a line in infinitely many ways — tilt it, shift it, graze one cluster or the other. Only one of those lines leaves the widest possible empty street between them, and choosing that one, rather than any of the others, turns out to be the entire idea behind the support vector machine.

The mental model

Many lines can separate two classes perfectly. Only one of them leaves the largest possible cushion of empty space on both sides — and choosing that one, rather than just any separating line, is what makes the classifier generalise.

A margin that can be faked

Start with the most literal way to measure how confidently a boundary separates a point: take the linear score w^Tx + b the boundary assigns to a point, and multiply it by the point's true label y ∈ {−1, +1}. Call this the functional margin.

Functional margin γ̂⁽ⁱ⁾ = y⁽ⁱ⁾ ( w^T x⁽ⁱ⁾ + b )

A positive functional margin means the point is on the correct side of the boundary; a larger one seems to mean the classification is more confident. But the quantity has an obvious flaw: multiply w and b both by 10, and every functional margin also multiplies by 10 — with the decision boundary itself completely unchanged, since it is defined by where w^Tx + b = 0, and that equation's solution set does not move when you rescale both sides. The functional margin can be made arbitrarily large just by rescaling the parameters, so it is not yet the right quantity to maximise.

Normalising it into an actual distance

The fix is to divide out exactly the rescaling freedom that broke the functional margin — divide by ‖w‖. What remains is the geometric margin: the actual perpendicular distance from the point to the decision boundary, a quantity that does not change no matter how w and b are rescaled together.

Geometric margin γ⁽ⁱ⁾ = y⁽ⁱ⁾ ( (w/‖w‖)^T x⁽ⁱ⁾ + b/‖w‖ )

Every term in the functional margin reappears here, just divided through by ‖w‖. That single division is the whole fix: w/‖w‖ is a unit vector, so the geometric margin measures a real distance in the input space, in the same units as x itself, invariant to how the boundary's parameters happen to be scaled.

Figure 7b The geometric margin is a measured length, not a score — the perpendicular distance from the point to the boundary, taken along the direction of w.

The optimal margin classifier

With a well-defined distance in hand, the objective becomes concrete: among all boundaries that separate the two classes correctly, find the one whose smallest geometric margin — the distance to the nearest point of either class — is as large as possible. Maximising the worst-case margin, rather than the margin of any single point, is what "widest street" means precisely.

That objective can be rewritten as an equivalent, and much more convenient, constrained optimisation problem. Fix the functional margin of the closest point to be exactly 1 — a choice that is always achievable by rescaling w and b, since scaling them together scales every functional margin uniformly — and then simply minimise ‖w‖ subject to every point meeting that functional-margin-1 threshold.

The optimal margin classifier min_w,b (1/2) ‖w‖² subject to y⁽ⁱ⁾ ( w^T x⁽ⁱ⁾ + b ) ≥ 1 for all i

Why does minimising ‖w‖ under a functional margin fixed at 1 amount to maximising the geometric margin? Because with the functional margin pinned at 1, the geometric margin of the closest point is exactly 1/‖w‖ — so making ‖w‖ as small as possible is the same computation, wearing a different name, as making 1/‖w‖ — the geometric margin — as large as possible. The 1/2 and the square are there purely for convenience in the calculus that follows; they do not change where the minimum sits.

Plate 7.1 Drag w and b over a fixed scatter and watch the widest-street objective turn into a number: the widget computes every point's functional margin live, marks whichever one is closest, and reports the geometric margin — the closest point's functional margin divided by ‖w‖ — and twice that as the street width. Push the sliders until the closest functional margin lands at exactly 1, and these become the canonical 1/‖w‖ and 2/‖w‖ that the optimal margin classifier is built to maximise.

Why the widest street generalises

The title's claim is not decoration; it names the actual mechanism. A boundary that barely squeezes between the two classes — a narrow street — is one where a small perturbation of the training data, or a new point drawn from a slightly different sample, can easily land on the wrong side. A boundary with the widest possible margin has to be pushed much further before that same small perturbation causes a misclassification. Robustness to resampling is exactly what generalisation to unseen data requires, and margin width is a direct, geometric proxy for that robustness.

Figure 7a Several faint candidate boundaries separate the two clusters correctly. The solid line is the one with the widest equal margin on both sides — touching the nearest point of each class and pushed no closer to either than it has to be.

What's left unsolved, and where it's solved next

As stated, the optimal margin classifier has n + 1 free variables — the n components of w plus b — and can in principle be handed directly to a general convex optimiser. That direct approach is called solving the problem in the primal.

Lecture 8 takes a different route: it rewrites this exact optimisation problem, via Lagrangian duality, into an equivalent dual problem. That rewriting is not merely an alternative algorithm for the same answer — it exposes a structural property of the optimal solution that changes how support vector machines are actually used in practice, and opens the door to classifying in spaces the algorithm never has to visit explicitly.

Read the primary source

CS229 Lecture Notes — Support Vector Machines. The functional/geometric margin derivation and the primal formulation.
CS229 SVM summary — aman.ai. A cross-verified transcription of the same material.

❦