Volume IV · Change the Mathematics

The Whiteboard Problem

A meeting room with one whiteboard and no cloth. Everything anyone has ever written is still up there, one layer of ink over another, and the board is exactly the size it was this morning. That is the memory you get when you take the softmax out of attention — and it is a bargain worth understanding before you decide whether to accept it.

The concession Volume III would not make

Every mechanism in the previous volume made the same promise and kept it. Sliding windows, attention sinks, the learned routers of NSA and MoBA — each one changed which pairs of tokens get compared, and none of them changed what happens once a pair has been chosen. The softmax stays. The scores it normalises are exact. Somewhere in the machine there is still a cache of keys and values, and the only question those lectures asked was how much of it a given query is allowed to touch.

This volume breaks the promise. It asks what attention would look like if the softmax were simply removed — not approximated, not masked, removed — and the answer turns out not to be a broken transformer. It is a recurrent neural network. It has been folded inside the equation since 2017, and the softmax is the fold.

One pair of brackets

The obstruction is a pair of brackets, and it repays a minute of close reading. Matrix multiplication is associative: given three matrices you may multiply the first two or the last two first, and the result is identical. In attention you are multiplying three things — Q, Kᵀ and V — so in principle the choice is yours. It is not yours, because a nonlinear function sits between the first product and the second. The softmax is applied to the n × n score matrix, which means that matrix must be built, in full, before V is touched at all.

Katharopoulos and colleagues asked, in 2020, what happens if you replace the exponential kernel with something factorisable: a feature map φ applied separately to queries and to keys, so that the similarity between a query and a key is just φ(q) · φ(k). The scores are no longer a softmax over the row. In exchange, nothing nonlinear separates the three factors, and the brackets move.

The left grouping builds an n × n matrix and then collapses it. The right grouping never builds it. The inner term φ(K)ᵀV is d × d — its shape is a property of the model, not of the input — and it is a sum over tokens, which means it can be accumulated as the tokens arrive rather than assembled at the end. Attention becomes a recurrence.

Written that way — with φ folded into the letters for legibility — the whole mechanism is two lines. Each token contributes an outer product of its value and its key to a running matrix; each query reads the matrix by multiplying through it. Training cost falls from O(n²·d) to O(n·d²), which crosses over the moment the sequence is longer than the head is wide. Generation is the more dramatic change: the KV cache goes from O(n) to O(1). There is no cache. There is one matrix per head per layer, the same size at token ten as at token ten million.

Which φ you choose is a research question of its own, and a productive one — the Performer line derives a feature map whose inner products approximate the softmax kernel in expectation, recovering some of what was given up. But the structural claim does not depend on the choice. Any factorisable similarity buys you the brackets, and the brackets buy you the recurrence.

The equation this volume is variations on

Before going further it is worth writing down the general form, because everything in the next two lectures is a substitution into it. A fixed-size memory S, a rule for what carries over from the previous step, and a rule for what the current token writes:

Linear attention is the simplest possible filling-in: f is the identity — the old state passes through untouched — and g is the outer product vkᵀ. Lecture 9 is a sequence of better choices of f. Lecture 10 is about how many layers you dare build out of any of them. Keep this box in mind; the rest of the volume refers back to it.

The whiteboard problem

Now the flaw, which is the lecture's title. With f the identity, the state only ever adds. Write a value for a key you have used before and the old association does not go anywhere — it is still in the matrix, underneath, contributing to every future readout for that key and for every key that resembles it. There is no subtraction anywhere in the update. Associations accumulate, and because they accumulate in a matrix of fixed size, they accumulate on top of each other.

That is the whiteboard with no cloth. Nothing about the board changes as the meeting runs on; only the ink gets denser, until a fresh line written across the middle is indistinguishable from the four lines already under it. The failure is not a crash or a spike in memory. It is a slow, quiet loss of precision that grows with the length of the sequence — exactly the regime the whole design was meant to serve.

What you actually traded

It is easy to present linear attention as a free efficiency win, and it is not one. A softmax cache is lossless random access. The key and value of the four-hundredth token are still sitting there, bit for bit, at the four-millionth; a query that matches them retrieves them exactly. A fixed-size state is a lossy summary. It cannot be otherwise — an arbitrary number of key–value pairs are being folded into a d × d matrix, and past some point the pigeonholes run out.

The canonical demonstration of the difference is needle-in-a-haystack retrieval: bury one specific sentence in a very long document and ask for it back verbatim. That task is not a quirky benchmark artefact. It is precisely the capability at issue — exact recall of a single earlier token, which is what an archive gives you and what a summary does not. When you read a hybrid architecture in Lecture 10, this is the deficit its full-attention layers are there to cover.

Two proofs and one bridge

The obvious objection to all of this in 2020 was that it might simply not work at scale. RWKV is the long-running answer. Begun as a community project and published in 2023, it is an attention-free language model trained to sizes that were then respectable and evaluated against ordinary transformers rather than against toy baselines. Its mechanism is a per-channel exponential decay: every dimension of the memory carries its own half-life, so the state fades rather than accumulating indefinitely. That is already a partial answer to the whiteboard problem, arrived at independently and years before the tidy formulation of Lecture 9 existed.

It is worth noting where that line ended up. RWKV-7, released in 2025 under the name Goose, converged on the delta rule — the update that the academic linear-attention literature reached from the other direction. Two traditions, one destination. Hold that thought until the next lecture.

RetNet, from Microsoft in 2023, deserves a paragraph for a different reason: it is the clearest statement that "train like a transformer, infer like an RNN" is a choice of factorisation and not a different model. RetNet defines three computational modes — parallel, recurrent, and a chunkwise hybrid of the two — that compute the same function. You train in the parallel mode because GPUs like it; you generate in the recurrent mode because it costs O(1) per token; you use the chunkwise mode for long-sequence training because it is the practical compromise. The insight generalises to every architecture in this volume.

RetNet's own forgetting mechanism was a fixed, non-learned decay — a constant rate of fading, chosen in advance and identical for every input. It was not enough, and the story of why not is the whole of the next lecture.