Volume VII · Reading the Reports

Reading a Deep Learning Paper

Nearly every fix in this series arrived with no clean proof behind it — only a result, and a plausible story for why it might work. Reading this era's papers well means noticing exactly where the story stops and the evidence starts.

A pattern this whole track has quietly followed

It is worth naming, now that fourteen lectures have gone by, the pattern this track has been living inside without saying so directly. The sibling Machine Learning Foundations track proves things: VC-dimension bounds on generalisation, the duality that makes an SVM a convex optimisation problem, the EM algorithm's convergence guaranteed by Jensen's inequality. Each of those results has a proof you could, in principle, check line by line. Almost nothing in this track has that.

Batch normalisation's own paper offered internal covariate shift as the reason it works — a theoretical framing that later work has seriously contested, even while the empirical result, that batch-normalised networks train faster and more stably, has never been in doubt. Dropout's ensemble-of-thinned-networks interpretation is elegant and genuinely useful for building intuition, but it does not fully explain everything dropout does in practice. ResNet's skip connection fixes the degradation problem dramatically and reliably — the "it's easier to learn an identity mapping than the full one" story is a useful intuition for why, not a formal proof of why deep plain networks degrade in the first place.

The reading discipline this implies

The practical consequence is a habit worth carrying past this course. When a deep learning paper makes a claim, separate what it measured from what it asserted. A benchmark number, a training curve, an ablation table — those are measurements, and they are the part of the paper that has held up best across this track's history. A story about why the measured effect happens — a mechanism, an analogy, an appeal to some other quantity going up or down — is a different kind of claim, argued rather than demonstrated, and it is exactly the part that later papers have gone back and revised.

This track's own papers are unusually good training ground for that habit, precisely because so many of them get the empirical mechanism right while their own stated theory for why it works has aged less well. Reading BatchNorm, Dropout and ResNet with that separation in mind is the same reading discipline the LLM Attention track's closing lecture applies to the field's frontier releases — treat a paper's number as more solid than a paper's story, and date the story.

Everything in this series is somebody else's foundation

None of this track's fixes stayed inside it. Word2vec and GloVe, Volume V's answer to turning a word into a vector, are the direct ancestor of every embedding lookup in the LLM Attention track — that whole other course begins exactly where this one leaves a token as a learned vector. The residual connection from Volume II is not a historical curiosity in the Transformer era; it is load-bearing inside every single block of every Transformer this site discusses, still doing the same job it was invented to do here: giving gradients a direct path backward through depth.

The variational autoencoder's ELBO, from two lectures ago, shares its core move — optimise a tractable lower bound because the true objective cannot be computed directly — with the Machine Learning Foundations track's own derivation of the EM algorithm from Jensen's inequality. And the GAN-to-diffusion handoff from the previous lecture is the exact same historical turn the Diffusion Models track opens with, told from the generative-adversarial side rather than the denoising side. Four volumes of this track, four separate load-bearing exports to the rest of the site.

The thesis this series opened with

Lecture 1 argued that depth is not free — that a naive deep network, more layers and nothing else, gets strictly worse, and that every mental model since has been an answer to some specific version of why. Fifteen lectures later that thesis has not softened; if anything it has sharpened. None of what made deep networks trainable was derived from first principles and then verified. It was discovered by trial against a failure that was actually observed — vanishing gradients, degrading accuracy, an unstable loss surface — and kept because it worked, with the explanation arriving afterward and, in several cases, aging worse than the fix itself.

Reading this track's papers as measurement-plus-story, rather than as settled theory, is the more honest way to hold everything in it. The empirical price was real and it was paid; the story attached to any one receipt is worth checking before you repeat it.