Volume IV · Change the Mathematics

The Truce

Open the architecture card of almost any model shipped in the last two years with a very long context, and the same silhouette appears: a tall column of cheap recurrent layers, and then — every third layer, or every ninth, or every twelfth — one layer that does the old expensive thing exactly. Nobody won the argument. Everybody divided the work.

Why purity lost

The case against a pure fixed-state stack is the one Lecture 8 made and Lecture 9 could only soften. A d × d state is a summary. It is a very good summary — with a learned gate and a delta-rule update it holds a great deal, and it holds it at constant cost — but it is a summary, and there are tasks for which no summary is an acceptable substitute. Quote this clause back to me verbatim. What was the second figure in the third table. Which variable did that function take on line four hundred. These are not exotic requests; they are most of what people do with a long context.

The case against a pure full-attention stack is the cache. Every one of those layers must remember every key and value it has ever seen, and the bill grows with the product of depth and length. The observation that makes hybrids work is that these two facts do not have to be traded off at the level of a whole model. They can be traded off layer by layer.

So keep a few full-attention layers and make the rest cheap. Exact recall survives, because somewhere in the stack the original keys and values are still on file. The cache shrinks in proportion, because it is now paid for by only those few layers. A model with one full-attention layer in four carries roughly a quarter of the cache of an all-full stack of the same depth — and that ratio, rather than any clever compression, is where the memory saving comes from.

The first generation

The genre begins in 2024 with Jamba, from AI21: Mamba layers, attention layers and a mixture-of-experts feed-forward, at a ratio of roughly seven cheap layers to one attention layer. It was the first hybrid released at a size that made it a serious model rather than an experiment, and it established the template that everything below is a variation on.

Two contemporaries took the idea in instructive directions. Samba, from Microsoft, interleaves Mamba not with full attention but with sliding-window attention — two cheap mechanisms combined, one from Volume III and one from Volume IV, on the argument that the window supplies the local exactness the state is worst at, and the state supplies the long range the window cannot reach. Zamba, from Zyphra, keeps a single shared attention block and reuses it at several depths, so the model pays for one set of attention parameters and applies it repeatedly. Both are answers to the same question — what is the cheapest thing that still restores exact recall — and neither answer is obviously wrong.

What is actually in production

By 2025 the hybrid had stopped being a research position and become an engineering default. Nemotron-H, from NVIDIA, is a Mamba-2 stack in which only about eight per cent of the layers are attention. IBM's Granite 4.0, released in October 2025, runs nine Mamba-2 layers to every attention layer across an enterprise product line — which is a different kind of evidence from a benchmark table, because a support contract is attached to it.

Falcon-H1, from TII, changes the geometry rather than the ratio: instead of alternating Mamba-2 blocks and attention blocks down the depth of the model, it runs both in parallel within the same block and combines their outputs. Every layer is then both cheap and exact, in whatever proportion the combination learns. It is worth knowing about precisely because it breaks the mental picture the rest of this lecture builds — the column of alternating stripes is a convention, not a law.

The 2026 pairs

The two most-discussed recipes of the past year both settle on three cheap layers to one expensive one, and both make the expensive layer earn its place. Qwen3.5, released in February 2026 at 397B total parameters with 17B active, alternates three Gated DeltaNet layers with one gated full-attention layer, giving roughly a fourfold KV-cache reduction at a context of 262K tokens. The cheap layer is the equation that closed Lecture 9; the expensive layer is the equation that opened Lecture 1.

Kimi Linear uses the same three-to-one shape with different components: three KDA layers to one MLA layer. Its more interesting decision is what the MLA layers do about position — which is nothing. They carry no positional encoding at all. The argument is that the KDA layers already impose an order on the sequence through their own recency dynamics, so adding a second, explicit notion of position to the attention layers is redundant at best. Position is delegated entirely to the recurrence. That architecture was scaled up in Kimi K3 in July 2026. Once the recurrent layers carry position, the attention layers are free to be pure content lookup — a cleaner job than the one they had in Lecture 1.

Nobody knows the right ratio

Look down the table below and the ratios run from 7:1 to 9:1 to 12:1 to 3:1, without any published account of why one number rather than another. They are, for the most part, engineering judgements — defensible, arrived at through internal ablations, and not comparable across labs because no two of these models share a training recipe.

Ant Group's ring-linear work is the first systematic public study of what the hybrid ratio should actually be. That such a study was still novel in late 2025 tells you how much of this design space is convention rather than measurement — which is an opportunity if you are looking for one, and a caution if you are quoting a ratio as though it were settled.

Leave the lecture with two axes rather than a list. The first is what the cheap layer is: a sliding window, a Mamba-2 state, a Gated DeltaNet, a KDA. The second is how many expensive layers you keep, and where in the stack you put them. Volume III and Volume IV meet exactly here — Samba's cheap layer is a Volume III mechanism and Qwen's is a Volume IV one, and the surrounding architecture is otherwise the same shape. The two volumes are not competing answers. They are two catalogues you shop from when filling in the same slot.