Volume VI · Generating Before Diffusion

Compress, Then Reconstruct

Take a photograph, force it through a pipe narrower than the photograph itself, and ask for the photograph back on the other side. Whatever survives that squeeze is, by construction, the picture's most essential structure — everything else was too wide to fit.

The bottleneck as an argument

The basic autoencoder is one of the oldest ideas in representation learning, and its argument is almost embarrassingly simple. An encoder f(x) → z maps an input x into a bottleneck vector z whose dimension is deliberately smaller than x's. A decoder g(z) → x̂ maps that bottleneck back out to something the same shape as x. The two are trained together, end to end, to minimise the reconstruction error ‖x − x̂‖² — the squared distance between what went in and what came back out.

Nothing here is generative yet. There is no notion of sampling, no notion of a prior, no claim about what happens if you hand the decoder a z it has never seen. It is a compression argument: if the network can pass x through a pipe narrower than x and still reconstruct it with low error, then z must be carrying whatever information in x actually mattered for reconstruction, and discarding the rest. That is dimensionality reduction earned by training rather than imposed by a fixed formula, and it long predates the deep learning era as a general idea in neural networks.

From a point to a distribution

A plain autoencoder's bottleneck is a single point. That is fine for compression, but it is a poor foundation for generation: sample a random point near two encoded images and there is no guarantee the decoder produces anything coherent, because nothing during training ever asked the space between real encodings to behave well. The variational autoencoder — Kingma and Welling, "Auto-Encoding Variational Bayes," 2013 — fixes exactly that gap by making the bottleneck a distribution instead of a point.

The encoder no longer outputs z directly. It outputs two vectors, μ(x) and σ(x), the mean and standard deviation of a Gaussian, and z is drawn from that Gaussian:

A prior is placed over that same space — ordinarily the simplest possible choice, N(0, I) — and training pulls every input's encoded distribution toward that prior while still keeping enough spread to reconstruct x correctly. The result is a bottleneck space that is continuous and reasonably well populated everywhere near the prior, which is precisely what lets you sample a fresh z after training and expect the decoder to hand back something plausible.

The trick that makes it trainable

There is an obstacle hiding in that Gaussian, and it is worth dwelling on because it is a genuinely clever piece of engineering rather than a formality. Backpropagation needs a differentiable path from the loss back to every parameter that produced it. Sampling z ~ N(μ, σ²) is a random draw — and a random sampling operation has no gradient with respect to μ and σ at all. Without a fix, gradients simply cannot pass through that node, and the encoder cannot be trained by gradient descent through the sampling step.

The reparameterization trick sidesteps the obstacle rather than solving it head-on. Draw the randomness from a fixed, parameter-free source instead — a standard normal ε ~ N(0, 1) — and compute z as a deterministic function of μ, σ and that ε:

Now the randomness lives entirely in ε, which carries no learnable parameters and needs no gradient. The path from z back to μ and σ is ordinary multiplication and addition, and gradients flow through it exactly the way they flow through any other layer. The stochastic sampling node itself is never differentiated — it is simply moved outside the part of the computation that gradients need to traverse.

The objective: a lower bound, not the real thing

Training a VAE means maximising the evidence lower bound, or ELBO — a quantity that is provably no larger than the true log-likelihood of the data, and that stands in for it because the true likelihood cannot be computed directly.

The first term is a reconstruction term: how well does the decoder recover x from a sample z drawn from the encoder's distribution q(z|x)? The second is a regularizer: how far has q(z|x) drifted from the prior p(z), typically the standard normal? Push the reconstruction term up and the KL term down together, and you are pushing the ELBO up — and the ELBO is a lower bound on the actual quantity you would like to maximise but cannot evaluate in closed form.