Volume III · Remembering — Recurrent Networks

Three Gates and a Cell State

Picture a conveyor belt running the length of a factory floor, carrying a single crate from one end to the other, untouched, unless someone deliberately reaches in to add something to it or take something out. That belt, and the three hands stationed along it, is the whole idea this lecture explains.

A second pathway, addition instead of multiplication

Hochreiter and Schmidhuber published the Long Short-Term Memory cell in 1997, and its central move is the one the previous lecture was building toward: alongside the hidden state h_t, the LSTM keeps a second state, the cell state c_t, and updates it in a way that does not repeatedly multiply the same matrix through a saturating nonlinearity. The paper itself calls this pathway a "constant error carousel" — a state that can carry a gradient across many timesteps largely unchanged.

Three gates decide what happens to that state at every timestep. Each is a small neural network of its own, a sigmoid applied to a linear combination of the previous hidden state and the current input, producing a vector of values between zero and one that acts as a soft, learned switch.

Reading the gates as a decision

Read the four equations as one decision made at every timestep: the forget gate looks at the previous hidden state and the new input and decides, per dimension of the cell state, how much of the old value to keep; the input gate and its candidate decide what new information is worth writing in and how much of it to add; and the output gate decides how much of the (now updated) cell state to expose as the hidden state that the rest of the network, and the next timestep, actually sees. Nothing about the cell state itself is squashed through a matrix multiplication at every step — only the gates computing how much to add or erase are.

The GRU's simplification

Cho and colleagues introduced the Gated Recurrent Unit in 2014, as part of an encoder–decoder architecture for machine translation, and it keeps the LSTM's core idea — a gated, largely additive state update — in a cheaper form. The GRU merges the forget and input gates into a single update gate z_t, and eliminates the separate cell state entirely: the hidden state h_t now serves both roles at once. A second gate, the reset gate r_t, controls how much of the past hidden state feeds into the candidate activation.

Where this stood, before attention

For roughly two decades, some gated recurrent cell — an LSTM, or later a GRU — was the default component for anything sequential: language modelling, translation, speech recognition. The gate is what let a network finally carry information across dozens or hundreds of timesteps, and every one of those applications depended on exactly the mechanism this lecture derives.

That default eventually gave way to a different architecture, one that dispenses with recurrence altogether and lets every position attend to every other position directly. The sibling Modern LLMs track opens with exactly that mechanism. It is worth carrying this lecture's vocabulary into that one: attention is, in its own way, another answer to the same question — what should the network be allowed to remember, and how far back should it be allowed to reach.