Volume V · Respect the Silicon

Virtual Memory for Tokens

A hotel takes a booking for a meeting and is not told how many people will come. It reserves the ballroom. Four people arrive. Do that a hundred times a minute and the hotel is full while most of its chairs are empty — which is roughly how language models were served until 2023.

The room booked for a meeting of unknown length

A serving engine faces a scheduling problem with one missing number. A request arrives, its prompt is known, and its response length is not — the model will decide that as it goes, and it might be twelve tokens or twelve thousand. The cache for that request must be allocated before generation starts. The safe move, and for years the standard one, is to reserve a contiguous slot large enough for the maximum possible length and hope most of it goes unused.

Most of it does. The vLLM authors measured the consequence and found that 60–80% of KV cache memory was lost to fragmentation and over-reservation. The loss comes in two flavours worth naming separately, because they are the same two an operating systems course names. Internal waste is the unused remainder inside a slot that was sized for a response far longer than the one that arrived. External waste is the gaps left between slots — memory that is genuinely free but not contiguous, and therefore not usable by a request that needs a single unbroken run.

Pages, and a page table

PagedAttention, published by Kwon and colleagues in 2023, is the fix, and a computer-science student will recognise it before the first figure is finished. Stop allocating a contiguous region per sequence. Chop the KV cache into fixed-size blocks. Give each sequence a block table that maps its logical token positions to whatever physical blocks happen to be free. Allocate a new block only when the previous one fills.

That is paged virtual memory, applied to tokens. The blocks are pages, the block table is the page table, and the attention kernel is taught to follow the indirection rather than assume the keys and values of a sequence sit next to each other. External waste disappears entirely, because any free block will do. Internal waste shrinks to at most one partly filled block per sequence. The reported result is fragmentation waste below roughly 4%, against the 60–80% it replaced.

Note what has been paid for this. The kernel must now walk a table to find each block, which is real work that the contiguous version did not do. The design is a straightforward trade: a small, predictable cost per access against a large, structural saving in capacity. That the trade is worth making is exactly the conclusion operating systems reached about paging sixty years ago, for the same reason.

The bonus that falls out of the design

The best part was not the point. Once blocks are indirected, nothing requires two block tables to point at different physical blocks. Two requests that begin with the same prefix — the same system prompt, the same document, the same few-shot examples — can share the physical blocks holding that prefix, with copy-on-write applied at the moment their contents diverge.

This is the mechanism underneath two features that are now taken for granted. It is what makes prefix caching possible, so a long shared system prompt is computed and stored once and is nearly free across every concurrent request that begins with it. And it is what makes parallel sampling cheap: generating eight candidate continuations of one prompt shares the prompt's blocks eight ways instead of copying it eight times.

vLLM, the engine built around PagedAttention, became the de facto open-source serving engine, and the approach was adopted conceptually across the ecosystem — paged, block-managed KV cache is now the assumed design rather than a distinguishing feature.

The research tax

The second half of this lecture is about a different scarcity. Suppose you have an idea for an attention variant — a new mask, a positional bias, a windowing scheme. Before 2024 the path from idea to usable experiment ran through a hand-written CUDA kernel. Express your variant in plain PyTorch and it would be correct and unusably slow, because it would materialise the score matrix and forfeit everything the previous lecture bought. Express it as a fused kernel and you had signed up for weeks of low-level work per idea.

That is a tax, and taxes shape behaviour. It meant that the variants which got tried were the ones somebody was willing to fund a kernel for, and the long tail of small, plausible, cheap-to-describe ideas mostly went untried — not refuted, simply never run.

FlexAttention, introduced by the PyTorch team in 2024, removes it. You express your variant as a small score_mod function — given a score and its coordinates, return a modified score — or as a block mask, and the compiler fuses it into a FlashAttention-class kernel. The reported performance is roughly 90% of a hand-written FlashAttention-2.

Read that list against the syllabus. ALiBi is Lecture 14. Sliding windows are Lecture 5. Attention sinks are Lecture 6. Each of them was, at the time it was introduced, a piece of systems work as much as a piece of research. Each of them is now a short function that compiles into a fast kernel, and a student can try all three in an afternoon.

Tooling shapes which ideas are cheap enough to have

Here is an argument worth making explicitly, offered as a reading of the record rather than a proven claim. Attention research accelerated visibly through 2025 and 2026 — more variants, more hybrids, more ratios tried and reported. Part of that is money and competition. Part of it, plausibly, is that trying a new mask stopped costing a quarter and started costing an afternoon.

When an experiment is cheap, more experiments get run, including the speculative ones that nobody could justify staffing. The set of ideas a field explores is not determined by imagination alone; it is bounded by what is affordable to test. Tooling moves that boundary, and moving it changes which ideas get had at all. Volume V is called Respect the Silicon, and this is the second sense of it: the silicon constrains the mathematics you can run, and the tools constrain the mathematics you bother to imagine.

Which closes the volume. The last lecture returns to a thread that has run under every one of them — how a model knows where a token sits — and to the surprising 2025 finding that in some layers the best answer is to say nothing at all.