Volume VI · The 2025–2026 Frontier

When Diffusion Meets Language

Every mechanism in this course so far was built to denoise pixels. In 2025 the same machinery started denoising something else entirely — whole sequences of text tokens, refined in parallel rather than written one word at a time.

Writing versus sculpting a sentence

Every language model in the sibling LLM track generates strictly left to right: one token commits before the next is even considered, a process this course's readers will recognise as autoregressive generation. A diffusion language model inverts that commitment. It begins from a sequence that is fully masked or noised and denoises all positions at once, refining the whole sequence together across a handful of steps rather than one token at a time. What is given up is the autoregressive guarantee that each token conditions exactly on everything decided before it; what is gained is parallelism, and the ability to revise an earlier "draft" word once later context makes clear it was the wrong choice.

Autoregressive generation is writing a sentence left to right, committing to each word before you have seen the rest of the thought. Diffusion-LM generation is closer to sculpting the entire sentence out of a block of masked marble at once — every position refined simultaneously, early guesses correctable as the whole picture sharpens, no position more finished than any other until the process ends.

The academic proof: LLaDA

Nie, Zhu, You and colleagues' LLaDA, "Large Language Diffusion Models," trained a model entirely from scratch — both pretraining and supervised fine-tuning — using a forward masking process and a Transformer-parameterised reverse unmasking process, with no autoregressive component anywhere in the pipeline. The result the paper reports is the headline claim of this lecture's first half: LLaDA-8B is competitive with LLaMA3-8B on in-context learning benchmarks. That is evidence the diffusion paradigm is not a curiosity confined to images — it can match autoregressive quality at a comparable parameter scale on language.

The middle ground: block diffusion

Pure diffusion language models have one awkward weakness that pure autoregressive models do not: generating an arbitrary, unbounded-length sequence is trivial for a model that writes one token at a time and simply keeps going, and considerably less natural for a model whose native unit of generation is a fixed-length sequence denoised together. Arriola, Gokaslan, Chiu, Yang, Qi, Han, Sahoo and Kuleshov's Block Diffusion — an ICLR 2025 Oral — fixes exactly this by interpolating the two paradigms named in this lecture's title.

The construction: define an autoregressive distribution over blocks of tokens, where each individual block's own internal distribution — conditioned on all prior blocks — is generated by a discrete denoising diffusion process. Between blocks, the model behaves exactly like an autoregressive one: block two waits for block one to finish and conditions on it. Inside a block, the model behaves exactly like the diffusion-LM of the previous section: every position in that block is denoised together. Unbounded length is recovered because block generation can simply continue; parallelism within a block is retained because that is still a diffusion process.

The commercial proof: Mercury

Inception Labs' Mercury, "Ultra-Fast Language Models Based on Diffusion," is the commercial case that the parallelism this lecture has been describing translates into a real speed advantage. Mercury Coder Mini reports over 1,100 tokens per second — roughly ten times typical autoregressive throughput — at 25 milliseconds average latency, and ranked second on the Copilot Arena human-preference leaderboard at the time it was measured. Mercury Coder Small reports 84.8% on a fill-in-the-middle coding benchmark, ahead of Codestral 2501's 82.5%. Coding is a plausible place for a diffusion-LM's parallel-refinement style to earn a real advantage: a fill-in-the-middle completion already has context on both sides of the gap, which is exactly the setting where resolving several positions together rather than committing left to right pays off.

The research signal: Gemini Diffusion

Google DeepMind announced Gemini Diffusion in May 2025 as an experimental research prototype, not a production release. DeepMind's own stated framing is that iterative refinement — revising a draft rather than committing to it left to right — particularly suits coding and mathematics, domains where a model benefits from being able to correct an early mistake rather than being locked into it. An open-weights follow-on, a 26-billion-parameter model released under the name DiffusionGemma, marked the first time Google shipped a diffusion-based text generator as open weights.

Taken as a set, the four models in this lecture are themselves a small case study in disclosure — a theme the closing lecture takes up directly. LLaDA and Block Diffusion are full academic papers with complete methodology. Mercury discloses substantial benchmark detail as a commercial product marketing a real speed claim. Gemini Diffusion's public information is closer to an announcement plus a model card than a full technical report. The underlying idea is the same across all four; how much of the machinery each party chose to show is not.