Volume III · Serving It Back

Paging the KV Cache

Volume II was about training a model too big for one device. This lecture opens a different problem: a trained model, running real production traffic, wastes most of its GPU cycles anyway — not because the model is too slow, but because the software scheduling and storing its work is bad at it. Two papers, arriving from different angles, each fix one half of that waste.

The waste hiding inside a batch

Serve a large language model to many users at once and the obvious move is to batch their requests together, the same way training batches examples to keep a GPU busy. But an inference request is not a fixed-size unit of work the way a training example is — it produces output tokens one at a time, and different requests in the same batch finish at wildly different times, because some prompts need ten output tokens and some need a thousand. Schedule a batch as a whole and the entire batch is stuck until its slowest member finishes generating: fast requests that are done sit idle, holding GPU memory and blocking new requests from joining, purely so the batch stays intact as a unit.

Orca (Yu, Jeong, Kim, Kim, and Chun, OSDI 2022) names this waste and fixes it by changing the granularity of scheduling itself. Instead of scheduling whole requests, Orca schedules at the level of a single model iteration — one forward pass. After every iteration, any request that has just finished generating is pulled out of the batch immediately, and a new, waiting request can be slotted into the freed slot without waiting for anything else in the batch to finish. This is continuous batching: the batch's membership changes iteration by iteration rather than staying fixed until every member is done. Orca pairs this with "selective batching" — since a batch may now contain requests at different stages with different sequence lengths, only the operations that are actually compatible across that heterogeneous batch get batched together, rather than forcing an artificial uniform shape. Orca reports a 36.9x throughput improvement over NVIDIA FasterTransformer at equal latency, serving GPT-3 175B.

The waste hiding inside memory

Continuous batching fixes scheduling, but it does not fix a separate problem: how the KV cache itself is stored. Each request's KV cache grows token by token and its final length is not known in advance. The conventional approach reserves one contiguous block of memory per request, sized for some maximum possible length — and that reservation fragments badly as requests of different actual lengths arrive, run, and finish, leaving unusable gaps between allocations the way a poorly managed heap does.

vLLM's PagedAttention (Kwon et al., arXiv:2309.06180, SOSP 2023) applies the fix operating systems settled on for exactly this kind of fragmentation decades ago: virtual memory paging. Instead of one contiguous block per request, the KV cache is split into small, fixed-size pages that can be allocated, freed, and — critically — shared, the same way an OS pages physical RAM into and out of a process's address space without requiring contiguous physical memory. A request's KV cache becomes a list of page pointers rather than one reserved span, so pages free up immediately as a request finishes and are handed to the next request without leaving a fragment behind. The paper reports a 2-4x throughput improvement over the prior state of the art at equivalent latency, with the gain more pronounced on longer sequences, larger models, and more complex decoding patterns — exactly the conditions under which contiguous-block fragmentation is worst. Paging also enables KV cache sharing within and across requests: a system prompt shared by many concurrent requests can occupy one set of pages referenced by all of them, rather than being duplicated per request.

Two fixes, one system

It is worth being precise about which paper fixes which waste, because they attack the same symptom — low GPU utilization during serving — from genuinely different causes. Orca's target is request-level scheduling: work sits idle because the scheduling unit is too coarse. PagedAttention's target is memory-level fragmentation: work is blocked because the storage unit is too coarse. Neither fix substitutes for the other — a system with continuous batching but contiguous KV allocation still fragments memory; a system with paged memory but request-level scheduling still lets fast requests idle behind slow ones. vLLM itself, the system that introduced PagedAttention, combines both ideas: continuous batching at the scheduling layer, paged allocation at the memory layer.