Volume II · Seeing — Convolutional Networks

A Filter That Looks Everywhere the Same Way

Take a 5×5 patch of weights that has learned to fire on a vertical edge. A fully-connected layer would need to learn that same patch, from scratch, at every position in the image where an edge might appear. Convolution asks a simpler question: why not use the one patch everywhere?

What a fully-connected layer costs on an image

Consider a small colour image, 32 pixels square, three colour channels. A fully-connected layer connecting every input pixel to every output unit needs one weight per (input pixel, output unit) pair. With 32×32×3 = 3,072 input values, even a modest output layer costs millions of weights — and every one of those weights is specific to its exact position in the image. A weight that has learned to detect an edge in the top-left corner contributes nothing to detecting the same edge in the bottom-right; that corner has to learn its own copy, from its own gradient signal, independently.

Convolution replaces that per-position weight matrix with one small, shared filter. A 5×5 filter over a 3-channel image needs 5×5×3 = 75 weights — regardless of how large the image is — and those same 75 weights are reused at every spatial position the filter visits. The parameter count stops scaling with image size entirely; it scales only with the filter's own dimensions.

Translation equivariance

Sharing weights across positions has a consequence worth naming precisely: translation equivariance. Shift the input image sideways by some amount, and the output feature map shifts by that same amount, unchanged in every other respect. A filter that has learned to detect an edge in one corner of the image automatically detects that edge anywhere else it appears — the property transfers for free, because it is the same weights doing the detecting regardless of where in the image they happen to be looking.

A fully-connected layer has no reason to have this property. Nothing forces its top-left weights and its bottom-right weights to agree on what an edge looks like; they are independent parameters that happen to sit in the same layer. Convolution's weight-sharing is what manufactures the guarantee, not an incidental side effect of using small filters — it is the entire reason to use them.

LeNet-5: the first working demonstration

The architecture that first put this to work at scale was LeNet-5, introduced by LeCun, Bottou, Bengio, and Haffner in 1998 for document and digit recognition. LeNet-5 alternated convolutional layers with pooling layers, stacking a handful of them before a small fully-connected classifier at the end — establishing the basic template that every convolutional network in this volume follows: convolve to detect, pool to reduce, stack to build hierarchy. Lecture 3 is that stacking, pushed much further.

What this buys, restated

Two claims, and only two: parameter sharing collapses what would be a per-position weight matrix into one small, reusable filter, cutting the parameter count from something that scales with image size to something fixed by the filter's own dimensions; and that same sharing manufactures translation equivariance — a property no fully-connected layer has any reason to possess, and one that turns out to matter enormously for images, where an edge is an edge no matter where in the frame it sits.