Volume III · Serving It Back

Splitting Prefill From Decode

Lectures 6 and 7 all assumed one thing without saying so: that a request's prefill step — reading the prompt — and its decode steps — generating the output — run on the same pool of GPUs, and tried to schedule or cache around whatever conflict that caused. This closing lecture of Volume III asks a blunter question: what if prefill and decode simply did not share a pool of GPUs at all?

Two phases, opposite profiles

Every transformer inference request actually has two distinct phases hiding inside it. Prefill processes the entire input prompt at once, computing the KV cache entries for every prompt token in a single pass — a large amount of matrix multiplication over many tokens simultaneously, which keeps a GPU's compute units highly utilized. It is, in the vocabulary of Lecture 7, compute-bound. Decode then generates the output one token at a time, and each of those steps needs to read the full accumulated KV cache to produce just one new token — a small amount of arithmetic relative to the amount of memory that has to move, which is exactly the memory-bandwidth-bound situation Lecture 7 described as the reason naive decoding leaves so much GPU compute idle.

DistServe (Zhong, Liu, et al., arXiv:2401.09670, OSDI 2024) is built on the observation that these two phases are not just different — they want opposite things from the same hardware. Run them on one shared pool of GPUs and they interfere: a large prefill batch arriving can delay decode steps for requests already in progress, since prefill's compute-heavy pass competes for the same execution units decode needs for its latency-sensitive per-token steps; conversely, a GPU busy serving many concurrent decode steps can delay a new request's prefill. Each phase, forced to share, ends up tuned for neither.

Disaggregation: don't share the pool

DistServe's answer is to physically separate the two: a distinct pool of GPUs handles only prefill, a separate pool handles only decode, and when a request's prefill finishes, its KV cache is transferred over the network from the prefill worker to whichever decode worker will continue it. Each pool can then be resourced and parallelized for its own bottleneck rather than a compromise between two — the prefill pool tuned for raw compute throughput on large batches, the decode pool tuned for memory bandwidth and low per-step latency across many concurrent requests. The paper frames its objective around "goodput" — a throughput metric that also accounts for the rate at which requests are served within their latency targets, rather than raw throughput alone, since a system that maximizes total tokens per second while violating every request's latency target has optimized the wrong thing.

The volume's closing argument

Lectures 6 and 7 both took the shared pool as a given and tried to schedule or cache around the conflict it created — continuous batching schedules around request-level idle time, paging schedules around memory fragmentation, speculative decoding and RadixAttention squeeze more out of the sequential decode step and the KV cache respectively. This lecture's move is different in kind: it does not optimize within the shared-pool assumption, it removes the assumption. That is worth tying back explicitly to the vocabulary Volumes I and II built. Tensor parallelism split a layer's matrices across devices; pipeline parallelism split layers across devices; Lecture 5 added splitting by sequence position and by expert. Disaggregating prefill from decode is itself a form of splitting the work — just splitting by request phase rather than by any structural piece of the model. The axis is new, but the move — find the dimension along which two things are actually forced to compete, and stop forcing it — is the same move this whole volume has been making.