Volume I · Why Depth

What One More Layer Buys You

Two networks can own the same number of adjustable weights and still not be the same machine. One is wide and shallow, its units spread across a single broad layer. The other is narrow and deep, the same weights stacked into a tall composition of small steps. On paper they are equally large. In practice, for years, only one of them could be made to work at all.

An argument about efficiency, not magic

It is tempting to hear "deep learning" and picture depth as an unqualified virtue — as if stacking layers were simply pouring in more capacity, the way adding rooms makes a house bigger. The actual argument is narrower and more interesting. A shallow network with enough units can approximate almost any function you like; that much was already known before depth became fashionable. What depth buys is efficiency: a composition of simple functions can express something a shallow network would need an exponentially larger number of units to approximate. Each layer reuses the work of the layers beneath it, the way a written proof reuses earlier lemmas instead of re-deriving them from axioms every time.

That is the whole theoretical case, and it is worth stating plainly because the rest of this series will not return to it: depth is not a magic property of neural networks, and it is not obviously good. It is a claim about representational efficiency, purchased at the cost of a much harder optimisation problem.

The edge that sat unused

Here is the complication this whole series exists to resolve: that efficiency argument is a statement about what a deep network can represent, not about whether you can find the weights that make it represent it. For a long stretch, deep networks did not train better than shallow ones. Naive deep networks — more layers, nothing else changed — got measurably worse. Errors that in principle a shallow layer's worth of weights should fix instead grew, and adding depth made training harder rather than more powerful. Lecture 4 is entirely about this degradation problem and the single connection that fixed a large piece of it; it is named here so that when it arrives, you recognise it as the payoff to a debt this lecture is opening.

So the honest history is not "depth was discovered to be good." It is: the representational case for depth was made early, then sat theoretically true and practically unusable for years, because nobody had solved the problem of actually training a deep network. Everything that follows in this series is that solving, one failure at a time.

Three unsolved problems, three volumes

Stacking more plain layers runs into trouble along three separate fronts, and each one gets its own sustained treatment later in this series rather than a paragraph here. Volume II is convolution's answer to a parameter problem: a fully-connected stack of layers over an image needs a number of weights that scales with the image's size, and weight-sharing fixes that by reusing one small filter everywhere. Volume III is recurrence's answer to a memory problem: a network that processes a sequence one step at a time has to carry everything relevant to that sequence in a fixed-size state, and naive recurrence forgets almost immediately. Volume IV is the least theoretically satisfying and most practically essential of the three — the optimizers, normalisation schemes, and initialisation rules that, with no clean closed-form justification, simply made deep networks trainable.

What this track is, and is not

It is worth being precise about how this series relates to its sibling, the Machine Learning Foundations track. That track proves things: generalisation bounds, margin arguments, convergence guarantees for well-behaved hypothesis classes. This track is mostly engineering — a sequence of fixes to empirical failures that only appeared once networks got deep enough for the failures to show up. A degradation problem that only manifests past a certain depth is not the kind of thing a clean theorem catches in advance; it had to be observed, then patched.

Neither track's rigor substitutes for the other's. The Foundations track will not tell you why a 56-layer plain network trains worse than a 20-layer one — that is an empirical fact this track spends a full lecture on. And this track will not give you a generalisation bound — that is not the kind of claim any of its architectures were built to support. Read them as complementary, not competing, accounts of the same field.