A lecture series · Revised July 2026

The Great Mental Models of Deep Learning

Before there was a transformer, there was a decade of hard-won answers to a narrower question: how do you make a network with many layers actually train? Convolutions that share weights, gates that stop gradients from vanishing, optimizers that adapt per parameter, and tricks with no elegant theory behind them that simply, empirically, worked. This series is the machinery the sibling tracks assume you already have.

Fifteen lectures · Seven volumes · Primary sources throughout

Scroll slowly ↓

Depth is not free. Every architecture and every training trick in this series exists because a naive deep network — more layers, no other changes — gets strictly worse, and someone had to find out precisely why.

Convolution answered "why should every patch of pixels re-learn the same filter." LSTMs answered "why does a network forget the beginning of a long sequence." Adam and batch norm answered "why is this loss surface so hostile to plain gradient descent." None of these questions has a clean closed-form proof the way the Machine Learning Foundations track's derivations do — this is a series of empirical fixes to empirical failures, and knowing the failure is how you understand the fix.

I

The depth problem

Stacking more layers should only ever help. In practice, past a point, it makes training actively worse — not from overfitting, but from optimisation itself breaking down.

II

The memory problem

A network that processes a sequence one step at a time has to carry everything relevant in a fixed-size state — and naive versions forget almost immediately.

III

The training problem

A deep network's loss surface is not the friendly bowl a introductory gradient-descent example promises. Plain SGD alone rarely reaches a good answer.

Volume I

Why Depth

before the architectures, the argument for having any

1 What One More Layer Buys You Why a deep, narrow network can represent what a shallow, wide one cannot — and why that theoretical advantage took years to actually train. Depth

Volume II

Seeing — Convolutional Networks

1998–2015 · a filter that looks everywhere the same way

2 A Filter That Looks Everywhere the Same Way Convolution shares one small filter's weights across an entire image — parameter efficiency and translation equivariance from a single design choice. LeNet 3 Stacking Filters Into a Hierarchy AlexNet's ImageNet moment and VGG's "just add more 3×3 convolutions" scaling attempt — depth as a straightforward lever, for a while. AlexNet · VGG 4 When Deeper Stopped Working, Until It Didn't Very deep plain networks got worse, not just overfit. ResNet's skip connection gives gradients a direct path backward and fixes it. ResNet

Volume III

Remembering — Recurrent Networks

1991–2014 · a network that feeds itself, and forgets

5 A Network That Feeds Itself The vanilla RNN's recurrence, and why backpropagation through time causes its gradients to vanish or explode over long sequences. RNN 6 Three Gates and a Cell State The LSTM's forget, input and output gates protect a separate cell-state pathway so gradients can survive long sequences. The GRU simplifies it. LSTM · GRU

Volume IV

Making Training Actually Work

1986–2015 · the tricks with no clean proof, only results

7 The Optimizer Zoo Momentum, RMSProp, and Adam — per-parameter adaptive learning rates, derived, and why plain gradient descent alone was never enough. Adam 8 Normalizing the Internal Signal Batch normalisation and layer normalisation keep every layer's activations in a well-behaved range — why this stabilises training so much. BatchNorm 9 Turning Off Neurons on Purpose Dropout randomly zeroes activations during training — an ensemble of exponentially many thinned networks, trained as one. Dropout 10 Starting From the Right Place Xavier and He initialisation choose a starting weight scale so signal neither vanishes nor explodes before training even begins. Initialisation

Volume V

Words as Vectors

2013–2014 · the bridge to language

11 A Word Is Defined by Its Neighbours Word2vec's skip-gram and CBOW objectives make the distributional hypothesis computational — a word's meaning is the company it keeps. word2vec 12 Counting, Then Compressing GloVe factorises global co-occurrence statistics directly, contrasted with word2vec's local sliding-window approach to the same idea. GloVe

Volume VI

Generating Before Diffusion

2013–2014 · the generative models the diffusion track supersedes

13 Compress, Then Reconstruct The autoencoder's bottleneck as representation learning, and the variational autoencoder's reparameterisation trick that makes sampling differentiable. VAE 14 Two Networks, One Argument The GAN's generator-versus-discriminator game, mode collapse, and why an adversarial objective eventually lost out to diffusion's plain regression loss. GANs

Volume VII

Reading the Reports

closing the series

15 Reading a Deep Learning Paper How architecture papers from this era are written and cited, and how the fixes in Volume IV quietly became assumed background in every later track on this site. Synthesis