Volume I · The Original Machine

The Bill Comes Due

Watch a model answer a long document. The weights never move — the same file sits on the same card, token after token. Something else is growing, quietly, one row at a time, and by the ten-thousandth token it is larger than the model itself.

Two bills, itemised

Lecture 1 ended with two debts left unpaid. The first is arithmetic: the n × n grid has to be filled in, and it grows as the square of the sequence. The second is storage: every head keeps its own keys and values, and during generation all of them must be remembered. Both were invisible in 2017 because sequences were 512 tokens long. Neither is invisible now. This lecture writes each of them as a formula with slots you can put real numbers into, because the rest of the course is a set of strategies for making one of those numbers smaller.

Keep the two bills separate in your head. They are paid at different times, by different parts of the hardware, and the mechanisms that reduce one frequently do nothing at all for the other. Confusing them is the single most common way to misread an architecture paper.

The compute bill

Scoring every query against every key means n² dot products, each over d dimensions, so the score computation costs on the order of n²·d. The softmax and the weighted sum have the same shape. Nothing here is subtle: double the context and you quadruple the work.

This is the bill people quote, and it is the one that dominates when a model is reading — processing a prompt, training on a batch, ingesting a document. All of those touch many tokens at once, so there is real arithmetic to do and the chip's multipliers are busy. Volume III is entirely about not filling in most of that grid.

The memory bill

The second bill is easier to write down and harder to escape. For every layer, for every key–value head, for every token seen so far, you are holding a key vector and a value vector of dimension d_head, each entry occupying some number of bytes.

Learn this line. Every mechanism in Volume II is an attack on the n_kv_heads × d_head term in the middle of it — the only part of the product an architect can change without changing the depth of the model, the length of the context, or the numerical format. Lecture 3 shrinks the first factor by sharing. Lecture 4 replaces both factors with something smaller by compressing.

Put numbers in. Take a model with 80 layers, 8 key–value heads and a head dimension of 128, stored at BF16, which is two bytes per value. Then 2 × 80 × 8 × 128 × 2 = 327,680 bytes for every single token — roughly 320 KB. At a context of 128,000 tokens that is about 40 GB. Half an 80 GB accelerator, occupied by one sequence, before you have served anybody else. Batching does not amortise this the way it amortises weights: every concurrent request carries its own cache. That is why serving throughput collapses as context grows.

Why the cache exists at all

It is worth asking why we store any of this. A transformer generates one token at a time, and each new token attends to every token before it. Without a cache, producing token number n would mean recomputing the keys and values for all n − 1 predecessors from scratch — quadratic work at every single step, and cubic work over a full generation.

Caching removes that repetition. A past token's key and value never change, because the causal mask means nothing later can influence them. Compute each one once, keep it, and the per-step cost becomes linear. This is one of the cleanest optimisations in the whole system, and it is entirely correct.

It is also the origin of the problem. The cache is an optimisation that creates its own bill: we traded repeated arithmetic for permanent storage, at a moment in history when arithmetic was the scarce resource. Volumes II through V are, collectively, the invoice for that trade.

Bandwidth, not arithmetic

Now the fact that explains almost everything that follows. To produce one token, the model must read the entire key–value cache — every layer, every head, every past position — and pass it through the attention computation. There is barely any arithmetic to speak of: a single query row against a very tall stack of keys. The multipliers on the chip sit largely idle while the bytes arrive.

So decoding is memory-bandwidth-bound, not compute-bound. The time to produce a token is governed by how many bytes must move between memory and the processor, not by how many multiplications happen once they arrive. Every byte you remove from the cache is a byte you do not have to stream, on every token, forever. This single sentence is why Volume II is worth four lectures and why Volume V exists at all.

It also reframes what "efficient" means. An architecture that halves the arithmetic and leaves the cache untouched will not make generation meaningfully faster. An architecture that halves the cache and leaves the arithmetic untouched will. Read every claim of speed-up with that distinction in hand, and ask which of the two regimes — reading or writing — the authors measured.

Three answers, one formula

The three configurations below are not different theories of attention. They are three different values plugged into the same line of arithmetic, chosen by three teams facing the same bill at different points in time. Read them as answers, not as alternatives.

Set the widget above to each of them in turn before reading on. The arithmetic is more persuasive than the prose, and the next two lectures are much easier to follow once you have seen how far apart those three bars sit.