Volume VI · Generating Before Diffusion
Two Networks, One Argument
A forger and an inspector, locked in the same room, each getting better because the other one is. Neither is told the rules of art or the rules of authentication — they are only told to win, and the argument between them is the entire training signal.
The mental model
Instead of training one network to reconstruct its input, train two networks to fight each other — one trying to fool, one trying to catch the fake — and the equilibrium of that argument is a generator that can produce data indistinguishable from the real thing.
A game instead of a loss
Generative Adversarial Networks — Goodfellow et al., 2014, presented at NeurIPS — replace the reconstruction objective of Lecture 13 with something structurally different: a minimax game between two networks trained at the same time. The generator G takes random noise and produces a sample it hopes looks real. The discriminator D looks at a sample, real or generated, and tries to say which it is.
Each network's training signal comes entirely from the other. G is pushed to produce samples D classifies as real; D is pushed to correctly separate G's output from the actual training data. There is no ground-truth "correct image" anywhere in the loss for G — the only supervision it gets is whether it currently fools an adversary that is simultaneously getting better at not being fooled.
No chains, no approximate inference
It is worth stating plainly what GANs did away with, because it is easy to take for granted in hindsight. Generative models before this point typically leaned on Markov chain Monte Carlo sampling or on approximate-inference machinery of the kind Lecture 13's ELBO represents. A GAN needs neither. Both G and D are ordinary feedforward networks, the whole system is trained end to end with plain backpropagation, and the only new ingredient is the two-player objective itself. That simplicity — no sampling chains, no variational bound to approximate — was a genuine part of the appeal.
Mode collapse
The game does not always settle where you would want it to. GANs' best-known failure mode is mode collapse: the generator finds a narrow slice of the true data distribution that reliably fools the current discriminator, and keeps producing variations on that slice rather than covering the distribution's full diversity. A generator trained on a dataset with many distinct classes or styles can collapse to producing convincing samples of only one or two of them.
This is a widely documented, empirically established phenomenon rather than something traceable to one single founding paper — it is presented here as an established fact about how the adversarial game tends to behave, not as a claim attributed to a particular study. It follows fairly directly from the game's structure: G is only ever rewarded for currently fooling D, and nothing in that reward explicitly asks G to cover every mode of the real distribution.
Why diffusion displaced GANs
Dhariwal and Nichol's 2021 NeurIPS paper, "Diffusion Models Beat GANs on Image Synthesis" — already the opening reference of the sibling Diffusion Models track — is the empirical proof point for this lecture's closing turn. Its result is exactly what the title says: diffusion models surpassed a strong GAN baseline, BigGAN-deep, on sample quality and FID across multiple ImageNet resolutions.
It is worth being precise about what that paper does and does not claim. It reports the benchmark result — diffusion beating GANs on FID at stated resolutions — as an empirical finding. The framing offered in this lecture, that a diffusion model's plain regression training objective is far more stable than a GAN's adversarial minimax game, is this lecture's own synthesis of why that empirical result came about, grounded in the paper's numbers rather than a sentence quoted from the paper itself. Say that plainly: the FID comparison is the paper's; the stability explanation is this lecture's reading of it.
The same hinge, from the other side
The Diffusion Models track opens its own first lecture from exactly this historical hinge — a generative modelling field moving away from an adversarial two-network game toward a single network trained with a plain, stable regression loss. This lecture and that one describe the same turn from opposite ends.
A GAN's training signal comes from a moving target — a discriminator that is itself being updated at the same time — and that game can oscillate, collapse, or simply fail to converge in ways a fixed regression loss cannot. A diffusion model, by contrast, is trained to predict a fixed, well-defined target — the noise added at a given step — with an ordinary supervised loss. There is no adversary and no moving target. That structural difference is the throughline connecting this lecture to the next track on this site.
| Family | Training signal | Known weakness |
|---|---|---|
| VAE (Lecture 13) | ELBO — a lower bound, optimised directly | often blurrier samples |
| GAN (this lecture) | a two-player minimax game | mode collapse, unstable training |
| Diffusion (sibling track) | plain regression against added noise | many sampling steps at inference |
Read the primary source
- Generative Adversarial Networks — Goodfellow et al., 2014.
- Diffusion Models Beat GANs on Image Synthesis — Dhariwal & Nichol, 2021.