Volume IV · Making Training Actually Work

The Optimizer Zoo

A ball released at the top of a narrow, curving valley does not roll straight to the bottom. It bounces from wall to wall, wasting most of its motion sideways, unless something gives it a memory of which direction it has already been travelling. That memory is the entire subject of this lecture.

The mental model

Plain gradient descent treats every parameter identically and remembers nothing between steps. Every optimizer in this lecture is a different answer to "what should we remember, and should every parameter move at the same rate."

Momentum — remembering direction

The oldest fix long predates deep learning. Boris Polyak described the "heavy ball" method in 1964, decades before arXiv existed to host it — this one is cited by its journal record alone. The idea is to accumulate a velocity from past gradients, so the optimizer is not told the descent direction fresh at every single step, but carries forward a fraction of where it was already heading.

Momentum — the heavy ball update w_t+1 = w_t − η∇ℓ(w_t) + ρ(w_t − w_t-1)

The last term is the memory: a fraction ρ of the previous step's displacement, added on top of the current gradient step. Down the narrow floor of a valley, that accumulated velocity builds up and smooths out the sideways bouncing that plain gradient descent cannot avoid, because plain gradient descent has no notion of "already heading this way."

Figure 7a Plain gradient descent responds only to the local slope, so it bounces across a narrow valley on every step. Momentum's accumulated velocity cancels that sideways component over time and accelerates along the direction that keeps being reinforced — the valley floor.

RMSProp — remembering scale, per parameter

Momentum still moves every parameter by the same learning rate η. Geoffrey Hinton proposed a different fix in slide 29 of Lecture 6e of his 2012 Coursera course, Neural Networks for Machine Learning — and it is worth stating plainly that this was never formally published as a paper. Its only citation is that unpublished lecture, a fact confirmed both by the lecture slides themselves and by Sebastian Ruder's widely-cited survey of optimizers, which cites it the same way: (Hinton, unpublished, 2012).

RMSProp's mechanism is to divide each parameter's gradient by a running average of its own recent squared magnitude — a per-parameter adaptive learning rate. A parameter whose gradient has been large and volatile gets its steps shrunk; a parameter whose gradient has been small and quiet gets comparatively larger steps. Momentum remembers direction; RMSProp remembers scale, separately for every parameter.

Adam — both memories at once

Kingma and Ba combined the two ideas in 2015. Adam keeps a first-moment estimate — an exponential average of past gradients, exactly momentum's memory of direction — and a second-moment estimate, an exponential average of squared past gradients, exactly RMSProp's memory of scale.

Adam — first and second moment estimates m_t = β₁ m_t-1 + (1 − β₁) g_t
v_t = β₂ v_t-1 + (1 − β₂) g_t²

Bias correction — compensating for a zero start m̂_t = m_t / (1 − β₁^t) v̂_t = v_t / (1 − β₂^t)

The parameter update θ_t = θ_t-1 − η · m̂_t / (√v̂_t + ε)

The bias correction exists because m_t and v_t are both initialised at zero — in the earliest steps, before the exponential averages have accumulated enough history, that zero initialisation biases both estimates toward zero. Dividing by (1 − β^t), a factor that starts small and approaches one as t grows, corrects exactly for that early bias without needing any change once training is well underway.

Figure 7b Plain SGD applies the same η to every parameter, the dashed line. Adam and RMSProp divide by each parameter's own running gradient scale, so frequently and violently updated parameters take smaller steps and quiet ones take larger ones.

What each optimizer actually remembers

Plain SGD remembers nothing between steps. Momentum remembers a direction. RMSProp remembers a per-parameter scale. Adam remembers both — which is precisely why it became, for a long stretch, the default choice almost nobody had to justify.

Reading the family tree

Laid end to end, these three are not competing inventions so much as one lineage, each step adding exactly one piece of memory that the previous one lacked. Momentum answers "which direction was I already moving in." RMSProp answers a different question entirely — "how volatile has this particular parameter's gradient been" — and applies its answer independently to every parameter rather than to the step as a whole. Adam is not a third idea; it is the previous two, run side by side, with a small correction for the fact that both running averages start at zero and need a few steps before they are trustworthy.

Plate 7.1 The family tree, run instead of read. Start on SGD and press step a dozen times — the path bounces wall to wall, exactly Figure 7a. Switch to Momentum at the same learning rate and watch the same steps smooth into the valley floor. Switch to Adam and check the per-axis step-size readout: the two axes end up taking the exact same step, even though the steep axis's raw gradient started twenty times larger — that per-parameter invariance is Adam's whole trick, and it happens without touching a single per-parameter setting by hand.

Three optimizers, what each one remembers
Optimizer	What it remembers	Origin
Momentum	an exponential average of past gradients (direction)	Polyak, 1964
RMSProp	a running average of squared gradients (scale, per parameter)	Hinton, unpublished, 2012
Adam	both — first and second moment, bias-corrected	Kingma & Ba, 2015

Read the primary source

Polyak, "Some methods of speeding up the convergence of iteration methods," USSR Computational Mathematics and Mathematical Physics, vol. 4, no. 5, 1964. Predates arXiv — journal citation only.
Neural Networks for Machine Learning, Lecture 6e — Geoffrey Hinton, Coursera, 2012. RMSProp's only citation; never formally published.
Adam: A Method for Stochastic Optimization — Kingma & Ba, arXiv:1412.6980, ICLR 2015. Combines momentum and RMSProp with bias correction.
An overview of gradient descent optimization algorithms — Sebastian Ruder, arXiv:1609.04747. The survey that confirms RMSProp's unpublished status.

❦