Volume IV · Making Training Actually Work

Starting From the Right Place

Before a single training step happens, before any data has been seen, one decision has already been made that can decide whether the network ever learns anything at all: the scale of its initial random weights.

A failure that predates learning

It is tempting to think of a badly-trained network as a story about gradient descent — the wrong learning rate, too few steps, a bad optimizer. But some of the worst failures happen before the optimizer takes a single step. If the initial weight variance is too small, activations shrink layer over layer, and so do the gradients flowing back through them, until both are effectively zero before learning starts. If the variance is too large, activations blow up instead. Neither failure is about the training algorithm; both are entirely about the starting point.

Xavier: matching variance across a layer

Glorot and Bengio's "Understanding the Difficulty of Training Deep Feedforward Neural Networks" (AISTATS 2010) states the goal precisely: choose the initialization variance so that the variance of activations in the forward direction, and the variance of gradients in the backward direction, stays roughly constant across layers. The derivation assumes linear or tanh-like activations, and it lands on a variance scaled as roughly 1/fan_in, or, in the form usually implemented, 2/(fan_in + fan_out) — a compromise that keeps both the forward and backward pass balanced at once.

Why ReLU broke the derivation

Glorot's derivation assumes an activation that is, to first order, symmetric around zero — a reasonable approximation for tanh. The rectified linear unit is not that kind of function. ReLU zeros out roughly half of all activations — everything below zero — which halves the variance passed forward compared to a linear or tanh unit at the same input variance. A variance formula derived for a symmetric activation quietly starts underestimating the variance loss the moment the activation is ReLU.

He: correcting for the missing half

He, Zhang, Ren and Sun's "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification" (arXiv:1502.01852, 2015 — the same paper that introduces PReLU) derives the correction directly from that observation. Since ReLU halves the variance passed forward relative to a linear or tanh unit, He initialization uses a variance of 2/fan_in specifically to compensate for that factor-of-two loss.

The target is exactly the one Glorot set out: keep variance roughly constant across layers. What changes is the accounting — He initialization restores that same constant-variance property, but does so while correctly crediting ReLU's effect on the forward pass, rather than assuming the symmetric activation that Glorot's derivation was built around.

A one-time decision with a lasting effect

Initialization is unusual among the topics in this volume because it does nothing at all after the first forward and backward pass — every weight update from then on is the optimizer's doing. But a network that starts with vanishing or exploding signal may never recover, no matter how well-tuned the optimizer from Lecture 7 or the normalization from Lecture 8 is. Depth, in other words, does not just need an optimizer that copes with a bad loss surface and a normalization scheme that keeps activations well-behaved during training — it needs a starting point that does not already doom the signal before either gets a chance to help.