Volume V · Making It Affordable

Trading Convolutions for Attention

Every diffusion model in this course so far has been built on the same convolutional scaffolding, narrowing down and widening back out, with skip connections stitching the two halves together. It has quietly done all the work. This lecture asks what happens when that scaffolding is torn out and replaced with the one architecture the sibling track on language models has spent its opening volume on.

Patches, not pixels

Peebles and Xie's 2022 Diffusion Transformer, DiT, makes one architectural swap and follows it through with unusual discipline. Instead of the convolutional U-Net used by every model discussed so far, take the — already-compressed, per the previous lecture — latent and chop it into patches. Treat each patch as a token exactly the way Volume VII of the LLM track treats a chunk of a document, and process the resulting sequence with a standard Vision-Transformer-style stack: alternating self-attention and feed-forward blocks, nothing convolutional anywhere in the backbone.

Timestep and class or condition information — the "which noise level" and "which category" signals a U-Net would ordinarily fold in through additive embeddings — enter through adaptive layer normalisation instead, specifically the paper's adaLN-Zero variant, which lets the network learn to ignore the conditioning signal entirely at initialisation and grow into using it during training.

A scaling law, not a hunch

The paper's central empirical finding is worth stating precisely, because it is a stronger claim than "bigger is better." Sample quality, measured by FID, improves monotonically as the transformer's compute budget — measured in Gflops, via more layers, wider layers, or more tokens — increases. There is no point on the curve where adding compute stops helping or starts hurting; the relationship is clean across the range the paper tests.

That monotonicity is the contrast being drawn. U-Net scaling, across the diffusion literature this track has surveyed, was historically more ad hoc: wider here, deeper there, an extra attention layer at one resolution, decisions made by trial and by intuition rather than by a rule a practitioner could bank on in advance. A DiT's uniform stack of identical blocks removes most of those design choices — there is little left to tune besides how many blocks and how wide, and the paper shows that turning either knob predictably helps.

The headline number attached to this scaling law: DiT-XL/2, the largest configuration tested, achieves a state-of-the-art FID of 2.27 on class-conditional ImageNet at 256×256 — at the time, the best published result on that benchmark, and reached by a backbone with essentially no convolutional inductive bias at all.

The same story, told twice

This is, structurally, the same story the LLM track opens Volume I with: a simple, general-purpose architecture that scales predictably with compute displacing a more specialised, hand-tuned one. There, the transformer displaced the recurrent network. Here, the transformer displaces the U-Net. The specific inductive bias being discarded differs — recurrence in one case, convolution and its implicit locality assumption in the other — but the shape of the trade is identical: give up an architecture that encodes useful prior knowledge about the problem, in exchange for one that encodes almost none and instead earns its performance by being easy to scale.

Taken together with the previous lecture, this closes Volume V. By 2023, diffusion models had accumulated latent compression to make each step cheap and a scalable transformer backbone to make adding compute predictable — an architecture recognisably similar to what the LLM track spends its first six volumes building out for language. That resemblance is not a coincidence this course will let sit unremarked: the final volume takes up the argument that the two fields, having converged on similar machinery from opposite directions, are now close enough to actually merge.