Volume III · Serving It Back

Guessing Several Tokens at Once

Lecture 6 fixed two kinds of serving waste, both about how many requests share a GPU at once. This lecture is about a single request, generating tokens one at a time, and two different ways to make that one sequential process faster — by guessing ahead, and by not repeating work that was already done.

Why decoding is slow in the first place

Generating text one token at a time means one full forward pass through the model per token: predict a token, feed it back in, predict the next one, and so on. Each of those passes is memory-bandwidth-bound rather than compute-bound — the GPU spends most of its time moving the model's weights and the growing KV cache through memory, not doing arithmetic on them — so a huge amount of the GPU's raw compute capacity sits unused during ordinary decoding. The sequential dependency is the obstacle: token N+1 cannot be predicted until token N exists, so naive decoding cannot parallelize across positions the way training, which sees the whole target sequence at once, can.

Draft, then verify in parallel

Speculative decoding breaks that dependency by decoupling proposing tokens from checking them. Two papers arrived at essentially the same idea independently and near-simultaneously in 2022–2023: Leviathan, Kalman, and Matias (arXiv:2211.17192, ICML 2023) and Chen, Borgeaud, Irving, Lespiau, Sifre, and Jumper at DeepMind (arXiv:2302.01318). Both describe the same mechanism. A small, fast "draft" model — cheap enough that running it several times costs far less than one pass of the large model — proposes a short run of candidate next tokens, one after another, the normal slow sequential way, but on a much cheaper model. The large "target" model then checks all of those candidates in a single forward pass, because verifying a fixed sequence of already-chosen tokens is exactly the kind of parallel-across- positions computation training does — it does not have the sequential dependency that generating those tokens in the first place did.

A rejection-sampling scheme decides which of the draft's guesses to keep: each candidate token is accepted or rejected by comparing the draft model's probability for it against the target model's probability, and the scheme is constructed so that the accepted sequence has exactly the distribution the target model would have produced generating token by token on its own. The moment a candidate is rejected, the target model's own distribution (adjusted for what was already rejected) supplies the replacement token, and drafting resumes from there. This guarantee is worth sitting with, because it is the surprising part: speculative decoding is not an approximation that trades a little quality for speed. It is provably identical output — same distribution the target model would have produced token by token, unmodified — achieved in fewer sequential large-model passes, because several candidate positions get checked together instead of one at a time.

Not recomputing what is already shared

Speculative decoding is a latency trick: do the same total amount of verification work, but compress it into fewer sequential steps. SGLang's RadixAttention (Zheng et al., arXiv:2312.07104, December 2023) attacks a completely different waste: recomputing or re-storing a KV cache that has already been computed for an earlier call. SGLang itself is a small domain-specific language for structured LLM programs — chained calls, control flow, multiple generation steps in one program — and RadixAttention is the KV-cache-reuse machinery underneath it: an LRU-cached radix tree data structure that indexes KV cache entries by the token sequence (the prefix) that produced them. When a new generation call shares a prefix with something already cached — a repeated system prompt, the same few-shot examples, an earlier turn of a multi-turn conversation — RadixAttention finds the matching node in the tree and reuses that cached KV state directly, computing only the suffix that is actually new. Crucially, this reuse is not limited to one conversation: the tree indexes prefixes across different requests too, so two unrelated requests that happen to share a system prompt both benefit from the same cached entry. The paper reports up to a 5x speedup on common LLM tasks from this reuse alone.