Volume IV · Change the Mathematics

The Eraser Trilogy

Three ways to clear a board. Let the ink fade on its own; scrub off the one line you are about to replace; or keep a cloth in each hand and do both. Every serious linear-attention architecture since 2020 is one of those three answers, and they arrived in that order.

Eraser one — ink that fades

Lecture 8 left a memory that could only be added to. The first repair is the one anybody would try: let the state shrink a little at every step, so that the past is continuously thinned out and no single association can dominate forever. Attach a coefficient α to the carried-over state, and the accumulated ink fades on a schedule. RetNet did this with a fixed rate, chosen once at design time and applied to every input alike.

Gated Linear Attention makes the rate a function of the data. The gate α_t is produced from the current token, so the model decides for itself, at each step, how much of its memory to keep — and because the gate is per-channel, it decides separately for every dimension of the state. Some channels are given long half-lives and behave like slow context; others are flushed almost immediately and behave like a scratchpad.

In the notation of the previous lecture, this is the first non-trivial choice of f: not the identity, but a learned, input-dependent contraction. The paper's other contribution is less quotable and more important — a chunkwise form that keeps the gated recurrence efficient on tensor cores, which is what makes a sequential update trainable at transformer speed.

The same forgetting, from another tradition

Mamba reaches input-dependent forgetting from an entirely different starting point. Its ancestry is state space models — a lineage out of control theory and signal processing, where a hidden state is advanced by a matrix and driven by an input, and the whole thing is a discretised linear system rather than a lookup table with the softmax removed.

The bars over A and B mark the discretisation; the subscript t marks Mamba's actual contribution, which is selectivity. In the state space models that came before, those matrices were fixed — the same filter applied to every input, which makes the system a convolution and makes it fast, but also makes it unable to treat one token differently from another. Mamba makes them functions of the input. The model decides per token what enters the state and what is flushed out of it, which is the same freedom the gate bought in the previous section, wearing different clothes.

Selectivity costs you the convolution, and with it the obvious parallel training path. The recovery is a hardware-aware parallel scan: the recurrence is computed in a tree rather than a loop, keeping the state in fast memory. The result is linear in sequence length for training, constant per token for decoding, and there is no KV cache anywhere in the architecture.

The Rosetta Stone

For about eighteen months these were two literatures. One spoke of feature maps, kernels and attention; the other of discretisation, scans and linear systems. They cited each other politely and had no common vocabulary.

State Space Duality closed the gap. The claim of Mamba-2 is that a selective state space model and a linear attention layer are the same mathematical object viewed from two sides: read it as a recurrence and you get the scan, read it as a structured masked matrix and you get the attention form. Neither view is a metaphor for the other. They are two factorisations of one computation, and — as in RetNet's three modes — you pick whichever one your hardware prefers for the job in front of you.

Teach this as the unification slide. It is why every paper after mid-2024 draws freely from both traditions, why the term "linear attention" and the term "SSM" are now used almost interchangeably at the frontier, and why the architecture in the next section can borrow a gate from one lineage and an update rule from the other without anybody remarking on it.

Eraser two — ink that is scraped off

Decay is a blunt instrument. It thins everything equally, which means the only way to remove a stale association is to fade the entire memory around it. What you actually want is the ability to replace one entry and leave the rest alone — a genuine key–value update to a fixed store.

DeltaNet does exactly that. Before writing a value for key k, it reads what the state currently predicts for that key and subtracts it. The projection k kᵀ identifies the component of the memory that answers to k; the term (I − β k kᵀ) removes it; then the new value is written in its place.

This is overwrite rather than accumulate, and the distinction is the whole lecture. Write the same key twice under the rule from Lecture 8 and you get the sum of two values. Write it twice under the delta rule and you get the second value.

The interpretation worth carrying away is this: the state is a tiny linear model, and the delta rule is one step of online gradient descent on the objective "predict v from k", with β as the learning rate. The layer is not storing memories; it is learning them, at test time, one token at a time. That framing has since become a small research programme of its own, and it recasts a long context not as a document to be searched but as a training set to be fitted.

None of which would matter if it could not be trained. A rule that reads the state before writing to it is sequential by construction, and sequential is fatal on a GPU. The paper's real contribution is an algorithm that parallelises the delta rule over the sequence length — which is what moved it from an idea in the literature to a layer you can put in a production model.

Eraser three — both hands

The two erasers solve different problems, so the obvious move is to use both. Gated DeltaNet, from Yang, Kautz and Hatamizadeh at NVIDIA, puts Mamba-2's decay gate in front of DeltaNet's targeted overwrite:

Decay handles graceful forgetting over time — the general dimming of context that is no longer relevant. The delta rule handles precise correction — this key, now, with this value. Neither substitutes for the other, and the combination is what shipped. Gated DeltaNet is the linear layer inside Qwen3-Next and Qwen3.5, and inside NVIDIA's Nemotron line. Of everything in this lecture, it is the equation most likely to be running on a server you talk to.

Kimi Delta Attention, published by the Kimi team in October 2025, is the same architecture with the gate refined. Where Gated DeltaNet applies one scalar forget rate per head, KDA applies a diagonal gate — one dial per memory channel, so each dimension of the state forgets at its own rate. That is the per-channel idea from §9.1 carried into the delta-rule setting, and it is the difference between a head having one clock and a head having sixty-four. In the hybrid model built around it, the team reports up to a sixfold decoding throughput improvement at a context of one million tokens, and a 75% reduction in KV cache.

If you want to read the mechanisms rather than the papers, the reference implementations of most of this lecture live together in the flash-linear-attention repository — GLA, DeltaNet, Gated DeltaNet and their chunkwise kernels, in a form you can put in a model.