Volume III · Skip the Work

The Bucket Brigade

A line of people stands between the river and the fire. Nobody walks the whole distance. Each one turns, hands a full bucket to a neighbour, and turns back — and the water arrives all the same. That is a sliding-window transformer, drawn without a single matrix.

Most of the grid is worth nothing

Print the attention map of a trained decoder and the first thing you notice is how empty it is. A few bright bands hug the diagonal, a column or two glows near the left edge, and the rest is a wash of near-zero weights that nonetheless cost exactly as much to compute as the bright ones. Softmax is scrupulous — it assigns a real number to every pair — but the distribution it produces is concentrated, and concentrated by a wide margin.

That observation opens two doors, and this course walks through them in order. The first, which is this volume, keeps the mathematics of attention untouched and simply declines to compute the cells that do not matter: exact softmax, over fewer pairs. The second, which is Volume IV, gives up on the pairwise grid altogether and replaces it with something whose memory does not grow. The first family is conservative and has shipped everywhere. The second is radical and is only now arriving.

The oldest answer to "which cells" is the simplest one: decide in advance, before the model has seen any data, and use the same shape for every input. A pattern chosen this way is cheap to implement, easy to reason about, and — as we shall see at the end of the lecture — permanently blind to content.

The ancestors

The first serious attempt was the Sparse Transformer (Child et al., OpenAI, 2019), which factorised the grid into two cheaper patterns applied in alternation: a strided one that lets a token attend to every kth position, and a local one that covers its immediate neighbourhood. Neither alone connects the sequence, but composed across two layers they do — any token can reach any other in O(√n) hops. The total cost falls to O(n·√n), which at the sequence lengths of the time was the difference between possible and not.

Then came the encoder pair that gave the field its durable vocabulary. Longformer and BigBird both settled on a local window plus a handful of global tokens — positions that everyone attends to and that attend to everyone, a small set of designated messengers wired to the whole sequence. BigBird added random links on top and, more importantly, proved that a sparse pattern of this shape preserves the theoretical universality of the full Transformer. You give up almost all the cells without giving up what the architecture can in principle represent.

The window in a modern decoder

Sliding-window attention reached the mainstream through Mistral 7B, which shipped with a window of w = 4096: each query attends to the last 4096 keys and no further. The arithmetic changes class, from O(n²) to O(n·w), and n·w is linear in n once w is fixed. That alone would be worth having.

The larger prize is the memory. A layer that can never look further back than w tokens has no reason to remember more than w keys and values, so its KV cache stops growing and becomes a rolling buffer: write the newest entry, overwrite the oldest, hold steady. After the four lectures of Volume II, spent watching that cache swell until it dwarfed the weights, a cache that simply refuses to grow is the headline result and not a footnote.

Depth carries the water

The obvious objection is that a token 20,000 positions back is outside the window and therefore lost. It is not, and the reason is the whole mental model. Layer one lets position n read positions n−w through n. Layer two reads the same span again — but every position in that span has already absorbed its own window, so what layer two actually sees is a summary reaching back 2w. Information moves forward one window per layer. With L layers the effective receptive field is L × w.

Hence the brigade. No individual in the line walks more than a few paces, and the fire still gets its water, because the distance is covered by the number of hands rather than the reach of any one pair. A model with 32 layers and a 4096-token window has a receptive field on the order of 131,072 tokens, assembled entirely out of short looks.

The interleave, not the window

Pure windowing turned out to be the wrong reading of the idea. What the field actually converged on is a mixture: mostly local layers, with a full-attention layer dropped in at intervals to restore exact long-range access. Gemma 3 uses five local layers to every global one, with a 1024-token window on the local layers, so the KV cache is dominated by the few global layers and the local ones cost almost nothing to keep. gpt-oss alternates dense layers with sliding-window layers of just 128 tokens — a window so short it would be indefensible on its own, and perfectly reasonable when a dense layer is never more than one step away.

Read those two designs together and the shape of the settled question appears. Nobody is arguing about whether to use local attention. They are arguing about the ratio: how many cheap layers can you afford between the expensive ones before the model stops being able to reach across its own context. A 1:1 alternation and a 5:1 interleave are two answers to one question, and the question is now a hyperparameter rather than a research direction.

What a fixed pattern cannot do

The limitation is structural and worth stating without softening. A fixed pattern is chosen before the model sees the input, and it is therefore blind to content. If the one token that decides the answer sits just outside the window, that layer cannot see it — not because the scores were low, but because the score was never computed. The pattern does not know what it is skipping. It skips by position, and position is a proxy for relevance that is usually good and occasionally catastrophic.

The interleave is a hedge against exactly this: the global layers are the escape route for the cases the windows would have missed. But a hedge is not a solution, and the honest next question is whether the model can be taught to choose its own pattern — to look at the past, cheaply, and decide for itself which parts of it deserve the expensive treatment. That is Lecture 7. First, in Lecture 6, we deal with a token that no window is allowed to throw away.