Volume IV · Making Training Actually Work

Turning Off Neurons on Purpose

Halfway through training a network, reach in and delete a random half of its neurons. Do this on every single step, a different half each time. On paper this looks like sabotage. In practice it is one of the cheapest, most effective regularizers ever found.

A network that cannot lean on anyone

A large enough network can memorize almost anything you show it, including the noise. Left alone, its units drift toward a comfortable division of labour: this neuron corrects for that neuron's known blind spot, that one only works because a third one upstream always fires first. This is co-adaptation, and it is a liability — the network has learned a set of fragile, interlocking crutches rather than features that stand on their own.

Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov's answer, published as "Dropout: A Simple Way to Prevent Neural Networks from Overfitting" (JMLR, 2014), is to make that dependence impossible to form. If a neuron can never be sure its neighbours will be present on the next forward pass, it cannot afford to rely on them. It has to become independently useful.

The mechanism, precisely

The rule is mechanical, not clever. During training, each unit — together with all of its incoming and outgoing connections — is independently zeroed with probability p, on every forward pass, for every training example. A different random mask is drawn each time. Nothing else about the architecture changes: the same weights, the same loss, the same backward pass, just applied to whichever units survived that particular draw.

An ensemble you never have to train

Here is the idea the mechanism is built to exploit. With n units each independently present or absent, the number of distinct sub-networks that could be sampled is exponential in n. Training with dropout is, in effect, training an exponential number of these thinned networks — but they are not separate models. They all share the same underlying weights, and any one weight is only ever updated on the steps where its unit happened to survive.

An ordinary ensemble would require training many separate networks and averaging their predictions at test time — expensive to train and expensive to run. Dropout gets a comparable effect from a single network, because the shared weights mean every sub-network's training also improves every other sub-network that shares those same weights.

What happens at test time

Averaging the predictions of every possible thinned sub-network exactly is not feasible — there are exponentially many of them. The paper's approximation is to use the single full network, with no units dropped, but with every weight scaled down by a factor of (1 − p).

The scaling matters because at training time a unit's expected output is already reduced by the chance it was dropped; at test time, with every unit present, that same unit would otherwise be firing at full strength into a downstream layer that learned to expect a weaker, thinned signal. Scaling by (1 − p) corrects for exactly that mismatch — a cheap stand-in for averaging an enormous ensemble, computed with a single forward pass through one network.

Dropout and the other kind of regularizer

The Machine Learning Foundations track's L1 and L2 penalties regularize by shrinking weight magnitude directly — a term added to the loss that punishes large weights, full stop. Dropout regularizes by a different route entirely: it never touches the loss's weight terms. Instead it forces robustness by preventing any single unit from being relied upon too heavily. Co-adaptation between neurons is disrupted because a neuron can never be sure its neighbours will be present on the next pass, so it has to encode something useful on its own.