Volume IV · Making Training Actually Work

Normalizing the Internal Signal

Send a signal down a long chain of amplifiers, each one built slightly differently from the last, and by the far end its volume has drifted somewhere no single amplifier was designed to handle. The fix used throughout this lecture is not to redesign the amplifiers. It is to reset the volume back to a known range after every single one.

Batch normalisation

Ioffe and Szegedy proposed batch normalisation in 2015, and its mechanism is exact: for each mini-batch, normalise each layer's pre-activations to zero mean and unit variance, computed across the batch dimension — that is, across all the examples in the current mini-batch, for one feature at a time. A learned affine transform, a scale γ and a shift β, is applied afterward, so the network can undo the normalisation entirely if that turns out to be the optimal thing to do.

The paper's own reported result is specific: the same accuracy on their benchmark was reached with fourteen times fewer training steps, and batch normalisation additionally enabled using higher learning rates with reduced sensitivity to how the weights were initialised in the first place.

Layer normalisation

Ba, Kiros and Hinton proposed layer normalisation in 2016, changing the axis the normalisation runs across. Instead of normalising across the batch dimension, layer normalisation normalises across the feature dimension, for a single example at a time. That single change has a consequence worth stating precisely: because it never depends on batch statistics, the same computation applies identically whether the network is training or being evaluated, with nothing to swap out at test time.

That property is also why layer normalisation applies naturally per-timestep inside recurrent networks, where a "batch" of hidden states at one timestep is an awkward thing to normalise across. It is worth noting explicitly that this same property is why Transformers — the architecture the sibling Modern LLMs track is built on — use LayerNorm, or its simplified cousin RMSNorm, rather than BatchNorm throughout: a Transformer processes variable-length sequences, and per-batch statistics are awkward to define consistently when sequence length itself varies example to example.

An honest close

Both techniques work extremely well empirically, and this is the point where the lecture's own mental model has to be taken seriously rather than smoothed over. Ioffe and Szegedy's original explanation for why batch normalisation helps — reducing what they called "internal covariate shift," the tendency of each layer's input distribution to shift as the layers before it keep updating — has been directly contested by later research, which has proposed other mechanisms entirely for the same empirical benefit.

The honest position, and the one this lecture takes, is to teach the mechanism and the empirical result plainly: both normalisations reliably make deep networks easier to train, faster to converge, and less sensitive to initialisation and learning rate — without overclaiming a settled theoretical account of exactly why.

What survives into later architectures

Whichever axis a given architecture normalises across, the underlying instinct — reset the signal to a known range at every layer, rather than trust that it will stay well-behaved on its own — turns out to be one of the few tricks from this whole volume that nobody has since found a reason to drop. It shows up wherever a network is deep enough that an unregulated signal could plausibly drift: convolutional networks, recurrent networks, and, in its LayerNorm and RMSNorm forms, the attention-based architectures the sibling Modern LLMs track builds on from its very first lecture.