Volume III · Remembering — Recurrent Networks

A Network That Feeds Itself

Pass a message hand to hand down a long line of people, each one half-remembering what they were told and half-mishearing it before repeating it to the next. By the far end of the line, almost nothing of the original sentence survives. A recurrent network is that line, and its hidden state is the message.

One equation, run in a loop

Everything a vanilla recurrent network does is contained in one small recurrence, applied again at every timestep of a sequence. A hidden state is produced by mixing the previous hidden state with the current input, then squashing the result through a nonlinearity:

Notice what is not there: no separate memory, no gate, no mechanism deciding what to keep and what to discard. The hidden state h_t is the network's entire account of everything it has seen so far, and it is produced by the same two weight matrices, W_hh and W_xh, at every single step. Run the recurrence a thousand times and you have used the same two matrices a thousand times over.

Backpropagation through time

Training a recurrent network means computing how the loss at the final timestep depends on the hidden state many steps earlier — and by the chain rule, that dependency is a product of every intermediate step's local derivative, one factor per timestep between the two. Because every step reuses the same weight matrix and the same activation function, this is not a product of many different terms. It is the same term, multiplied by itself, over and over.

Each tanh′ factor is at most 1, and typically far smaller away from the origin, since tanh saturates almost everywhere except a narrow band near zero. So the product on the right shrinks with every additional step, while the (W_hh)^T term is repeatedly applying the same matrix — whose effect over T applications is governed entirely by its dominant eigenvalue.

Vanishing, or exploding

Combine the two effects — the repeated matrix and the repeated derivative — and the outcome is decided almost entirely by one number: the dominant eigenvalue of W_hh, scaled by the local tanh derivative. If that scaled eigenvalue is less than one, every additional timestep multiplies the gradient by a factor below one, and the gradient shrinks exponentially in T. Push it back twenty or thirty steps and it is indistinguishable from zero — the network receives no signal at all about how an early timestep should have behaved differently.

If the scaled eigenvalue is greater than one instead, the same exponentiation runs the other way: the gradient grows exponentially in T, and training diverges, often visibly, as loss spikes or turns to NaN. Both failure modes come from the identical mechanism — a single quantity raised to a large power — which is why they are always discussed as a pair rather than as two separate bugs.

What a dozen steps buys you

Bengio, Simard and Frasconi made the same point rigorously in 1994: gradient-based learning in a recurrent network becomes exponentially harder as the dependency it must learn spans more timesteps, and this is a property of the architecture, not a symptom of a poorly tuned run. The practical consequence is blunt. A vanilla RNN cannot reliably learn dependencies spanning more than roughly a dozen or so timesteps — a pronoun referring back to a noun ten words earlier is already near the edge of what it can carry.

That limit is the whole reason the next lecture exists. If the problem is that a single pathway is repeatedly multiplied and squashed, the fix is to give the network a second pathway that updates by addition instead — and to let the network itself learn when to add, when to overwrite, and when to forget.