Volume II · Seeing — Convolutional Networks

When Deeper Stopped Working, Until It Didn't

Take a 20-layer plain convolutional network and a 56-layer version of the same design, trained on the same data. The deeper one should, at worst, match the shallower one — it could simply learn to leave its extra layers doing nothing. Instead its training error, not just its test error, comes out higher. Something is broken in the optimisation itself.

The degradation problem, stated precisely

He, Zhang, Ren, and Sun, at Microsoft Research, framed the problem exactly this way in the paper that introduced ResNet: as plain, non-residual networks are made deeper, accuracy first saturates, then degrades rapidly — and this is explicitly not overfitting. If it were overfitting, the deeper network's training error would stay low while only its test error suffered. What they observed instead was the training error itself rising for the deeper plain network. The network was failing to fit its own training data, not failing to generalise from it.

That is the detail that makes this a genuinely strange result rather than an expected one. A deeper network should, in principle, be at least as capable as a shallower one: give the extra layers weights that compute the identity function — pass their input straight through unchanged — and the deeper network reduces exactly to the shallower one, no worse. The extra depth costs nothing if it does nothing. What He et al. found is that, in practice, a stack of plain nonlinear layers has a hard time learning to approximate that identity mapping. The optimiser simply does not find it, even though it exists.

The residual connection

ResNet's fix is a single structural change. Instead of asking a stack of layers to learn a desired mapping H(x) directly, let the stack learn a residual, F(x) := H(x) − x, and produce the block's output as F(x) + x — the transformed path added back to the untransformed input, carried forward by a shortcut connection.

The relabelling looks cosmetic and is not. Learning F(x) ≈ 0 — the residual doing nothing — is a much easier target for a stack of nonlinear layers to hit than learning H(x) = x directly through those same layers. Driving a set of weights toward zero is a natural, easy-to-reach point for gradient descent; reproducing an exact identity mapping through several nonlinear transformations is not. The shortcut does not make the network more expressive — a plain stack could in theory already represent an identity mapping — it makes that particular, useful mapping easy to *find*.

Why this fixes training, not just representation

The mechanism is about gradients, not just forward representation. The shortcut connection gives the gradient a direct, additive path straight back through the network — an identity path, whose derivative with respect to its input is exactly one — running in parallel with the path through F. Even when F's local gradient is small, the total gradient reaching that block does not vanish, because the identity path contributes its share regardless. Depth stops being an obstacle to gradient flow, because every block has a route for gradient to pass through unchanged.

The empirical result matched the mechanism: ResNet trained cleanly up to 152 layers — eight times deeper than VGGNet, and at lower computational complexity — and took first place across ImageNet classification, detection, and localisation, plus COCO detection and segmentation, in 2015.

Where this idea resurfaces

The residual connection did not stay confined to convolutional networks. It is reused directly inside every Transformer block — the sibling series on LLM attention builds its entire architecture around a residual stream running past each attention and feed-forward sub-layer, though that series does not dwell on where the idea originally came from. When you meet it there, this is the lecture it traces back to: the same problem, a stack of layers whose gradient needs an unobstructed way home, and the same fix.