Volume II · Shrink the Memory

A Zip File for Memories

An archive that has run out of shelves has two remedies. It can throw boxes away, which is quick and irreversible. Or it can find that most of what the boxes contain is already implied by the rest, and store only the difference. The second remedy takes a cleverer archivist.

Compress, don't discard

Grouped-query attention saves memory by deciding in advance which heads may not have their own opinion. Multi-head Latent Attention, introduced with DeepSeek-V2 in 2024, starts from the opposite instinct. It keeps every head's key and value conceptually intact and refuses to store them in that form. A learned down-projection maps each token into a single low-rank latent vector — 512 dimensions in DeepSeek-V2 — and it is only that latent which enters the cache. Learned up-projections reconstruct the per-head keys and values when attention needs them.

The idea is straightforward enough that the interesting question is why it took until 2024, and the answer is that a naive implementation is worse than useless. Decompressing a latent into full keys and values at every step, for every layer, would replace a memory bill with a compute bill — and Lecture 2 established that decoding cannot afford either.

The trick that makes it free

The resolution is algebraic rather than computational. Attention scores are queries multiplied by keys; outputs are attention weights multiplied by values and then by the output projection. If the keys are themselves a fixed matrix applied to a latent, then that matrix can be multiplied into the query projection ahead of time. The same is true of the value up-projection and the output projection. Matrix multiplication is associative, and the up-projections are constants after training.

So the reconstruction never happens. The up-projection matrices are folded into the query and output projections once, offline, and at decode time attention reads the latent directly. You compress on the way in and you never decompress. That is the whole engineering achievement, and it is why MLA is a real inference win rather than a paper one.

The numbers

Return to the formula from Lecture 2. Under multi-head attention the cache holds 2 × n_heads × d_head values per token per layer. Under MLA it holds d_c + d_rope — the latent plus a small positional channel we are about to meet — which in DeepSeek-V2 is 512 + 64.

DeepSeek reported roughly a 93% reduction in KV cache against multi-head attention — and reported matching or beating MHA quality while doing it. That second half is the headline. MQA and GQA are trades: you name a quantity of quality you are willing to lose and you collect memory in return. A learned low-rank projection is not obviously a trade at all.

The RoPE wrinkle

There is a complication, and it is the most instructive moment in this lecture. Rotary position embedding encodes a token's position by rotating its query and key vectors by an angle that depends on where the token sits. The rotation is applied per position, which means it does not commute with the low-rank projection.

Follow the consequence. If you rotate before compressing, the position-dependent rotation is baked into the thing you cached; each cached latent then carries a rotation belonging to one specific position, and the up-projection can no longer be folded into the query projection as a constant. The algebraic trick of § 4.2 collapses, and with it the entire benefit. Compression and rotary position, implemented naively, are incompatible.

DeepSeek's answer is a decoupled RoPE. A small separate set of dimensions is set aside to carry position; those dimensions are rotated and cached alongside the latent, while the compressed content channel stays position-free and therefore foldable. Position moves to a side channel. The 64 dimensions in the formula above are precisely that channel, and they are the price of keeping the other 512 compressible.

This is the first appearance in production of what later became known as partial RoPE — the observation that not every dimension of every head needs to carry positional information, and that separating the ones that do buys architectural freedom elsewhere. Lecture 14 returns to it in its own right.

Why compression beats sharing

Both mechanisms in this volume rest on the same observation: keys and values are redundant. They differ in what they assume about where that redundancy sits.

Grouping assumes it is distributed evenly, in blocks you nominate before training. That assumption is baked into a hyperparameter and cannot adapt. A learned low-rank projection assumes only that keys and values lie near some low-dimensional subspace, and then finds that subspace from the data. Where the redundancy actually is becomes a question the training answers rather than one the architect guesses. That is the whole of the difference, and it is enough to turn a lossy trade into something close to a free one.

What it costs, and who paid it

Nothing here is free of consequence. MLA is substantially more intricate than grouping — to implement correctly, to fold, to fuse into a serving kernel, and to reason about when something goes wrong. Grouping is a change to a single dimension in a configuration file. Latent attention is a redesign of the layer, with a positional side channel bolted to it.

That complexity budget explains the adoption pattern. GQA remained the default for smaller open models well after MLA was published, because a team shipping a seven-billion parameter model gets most of the serving benefit for almost none of the engineering risk. MLA has been carried forward chiefly by the labs whose scale makes cache the binding constraint.