Volume II · Seeing — Convolutional Networks

Stacking Filters Into a Hierarchy

A single convolutional layer learns edges. Stack a second layer on top and it learns combinations of edges — corners, simple textures. Stack a third and those textures become parts: an eye, a wheel, a patch of fur. Depth, here, is composition made literal: each layer's vocabulary is built entirely out of the layer beneath it.

AlexNet's turning point

In 2012, Krizhevsky, Sutskever, and Hinton entered an eight-layer convolutional network — five convolutional layers followed by three fully-connected layers — trained on two GPUs against the full ImageNet ILSVRC-2012 dataset: 1.2 million images across 1,000 classes. It won the competition's classification task with a top-5 error of 15.3 percent, against 26.2 percent for the second-place entry — a margin so large it was not read as an incremental improvement but as a demonstration that this approach worked at all.

Three choices inside AlexNet mattered as much as its depth. It used ReLU activations in place of the tanh or sigmoid units that had been standard — ReLU does not saturate for positive inputs, so it trains substantially faster and keeps gradients alive further into a deep stack. It applied dropout in the fully-connected layers to control overfitting on a network large enough to memorise its training set. And it was trained on two GPUs simply because the network and dataset were too large for one — an engineering constraint as real as any architectural decision here.

VGGNet: depth via small, repeated filters

Two years later, Simonyan and Zisserman's VGGNet asked a more disciplined version of the same question: instead of a few large-kernel convolutional layers, what happens if you stack many small ones? VGGNet's networks ran sixteen to nineteen weight layers deep, built almost entirely out of 3×3 convolutions. Two stacked 3×3 filters cover the same receptive field as one 5×5 filter — the area of input each ultimately "sees" is identical — but the two-filter stack uses fewer total weights and inserts an extra nonlinearity between them, which the single larger filter cannot offer. VGGNet placed first and second in ImageNet 2014's localisation and classification tracks.

Edges into objects

The composition argument from Lecture 1 is easiest to see concretely in exactly this kind of stack. The earliest convolutional layer, working directly on pixels, tends to learn simple oriented edges and colour contrasts — the same handful of filter shapes recur across networks trained on entirely different datasets. The next layer takes those edge responses as its input and combines them: a filter that fires only when two particular edges meet at a corner, or when a small patch of edges forms a repeating texture. Neither layer was told to specialise this way; it falls out of what is available to build with at each depth.

Continue the stack far enough and the vocabulary keeps compounding: textures into parts — a wheel, an eye, a patch of fur — and parts into the whole objects a classifier ultimately has to name. Each layer's filters are defined entirely in terms of the layer beneath it, which is exactly the efficiency argument from Lecture 1 made visible: a shallow network attempting to detect "wheel" directly from pixels would need to relearn every edge and texture that composes a wheel, inside a single layer, for every object category that happens to contain one.

"Deeper" as a reliable lever

Read together, AlexNet and VGGNet tell a simple story: from 2012 to 2014, adding depth to a convolutional network reliably improved results, provided the engineering — GPUs, dropout, careful filter choice — kept pace. There was no hint yet of the wall the next lecture describes. Depth, for this brief window, behaved exactly as the naive intuition says it should: more layers, more hierarchy, better performance.