Volume III · Skip the Work

The Scout and the Army

An army does not march across unknown country with its whole weight forward. It sends one rider ahead, cheap to feed and quick over ground, and moves the heavy column only along the line the rider comes back and points at.

The trouble with deciding in advance

Every pattern in the previous two lectures was drawn before the model saw a single token. A window keeps what is near; a global token keeps what was designated; a sink keeps what happened to be first. All three are proxies for relevance, and all three fail in the same way — silently, on the cases where relevance and position come apart. The crucial sentence sits one position outside the window and is not merely down-weighted but never scored.

The alternative is obvious to state and hard to build: let the model decide. Score the past cheaply, pick the parts that matter for this particular query, and run full attention only there. The difficulty is that "pick" is discrete, discrete choices do not differentiate, and a selection procedure bolted onto a model trained with dense attention tends to select badly — the model was never taught to make its representations legible to a scout. Three systems, all from 2025, solved this the same way: by training the model sparse from the beginning, so that the selection mechanism and the attention it feeds grow up together.

NSA — three views, gated

DeepSeek's Native Sparse Attention, published in February 2025 and awarded best paper at ACL that year, gives each query three parallel branches and learns a gate over them. The first is compressed: the past is divided into blocks and each block reduced to a coarse summary, so that every query gets a skim of everything, cheaply. The second is selected: the top-n blocks by score are re-read at full fidelity — the pages worth turning back to. The third is a sliding window over the recent past, because local context is always wanted and there is no sense making the selector rediscover that fact on every query.

Skim everything, re-read what matters, and keep your finger on the current line. Two properties make it more than a good diagram. It is trained sparse from the start, so the selection is learned rather than imposed. And it is block-shaped, deliberately, so that the memory access pattern matches what GPU tensor cores are built to do — a token-scattered selection with the same theoretical cost would run far slower. NSA reports roughly an 11× decode speedup at 64K context.

MoBA — mixture of experts, applied to the context

Moonshot AI's Mixture of Block Attention, also from February 2025, arrives at a similar place from a different tradition. Mixture-of-experts routing normally selects among parameters: many feed-forward experts exist, each token activates a few. MoBA points the same machinery at the context. Chop the past into blocks, mean-pool each block's keys into a single gist vector, and let each query route itself to its top-k blocks by similarity to those gists.

Picture a bookshelf where every chapter has a spine label. The query reads the spines — which is cheap, because there are far fewer spines than pages — and opens three chapters. The design has an unusual and practically valuable property: because the parameters are entirely unchanged, a MoBA model can switch between sparse and full attention at will, sparse for long-context serving and full when the extra fidelity is worth paying for. The routing is a behaviour, not a weight surgery.

DSA — the scout that shipped

The version that reached production is DeepSeek Sparse Attention, in DeepSeek-V3.2. It replaces the block gist with something more literal: a small FP8 lightning indexer with very few heads, which scores every past token against the current query at a fraction of the cost of real attention. The top-k tokens are kept — k = 2048 — and full MLA attention runs on those alone. The scout reads the whole country; the army marches on two thousand positions.

Two things distinguish it from NSA. It selects at token granularity rather than block granularity, which is finer and therefore more precise about what it keeps. And it is retrofit-friendly, which matters more than it sounds: V3.2 was continued-trained from V3.1 rather than trained from scratch, by first distilling the indexer against the dense attention of the existing model — teaching the scout to agree with the army — and only then switching the model over to sparse operation. The commercial result is the cleanest evidence in this volume that the technique works: DeepSeek's long-context API pricing roughly halved.

Adoption is the proof

Papers are cheap and architectures are not; the honest test of a technique is whether another laboratory stakes a flagship on it. GLM-5, published by Zhipu in February 2026, adopts DSA on top of MLA — one laboratory's published attention mechanism appearing, essentially intact, in another laboratory's frontier model. That is a stronger signal than any benchmark table.

MiniMax's M3, from June 2026, makes the same bet with its own construction: MiniMax Sparse Attention, a block-level selection scheme on a GQA backbone with a great deal of kernel work underneath it. The claimed figures are 9.7× faster prefill and 15.6× faster decode at one million tokens. The asymmetry between those two numbers is itself instructive — sparsity pays most where the sequence is longest and the per-token work is smallest, which is to say during generation.

The lookup table, literally

Lecture 1 introduced attention as a soft lookup table, and the word "soft" was doing real work: nothing was retrieved, everything was blended, and the metaphor was a teaching device rather than a description of the implementation.

That is no longer quite true. DSA scores a corpus of past tokens with a cheap model, ranks them, keeps the best two thousand, and reads only those in full. Strip away the vocabulary and it is a retrieval system — a coarse ranker feeding a precise reader — living inside the attention layer and trained end to end with the model it serves. The metaphor has become the architecture. What Volume III cannot do, for all its ingenuity, is make the memory itself stop growing: every system here still keeps the keys and values of every token it might later select. Volume IV asks what happens if you refuse.