Volume VII · Neural Networks From First Principles

The Chain Rule, Applied Systematically

Every gradient in this track so far was derived once, by hand, in closed form, because the hypothesis class was simple enough to allow it. A neural network is a composition of as many differentiable layers as you care to stack, and no one is going to re-derive its gradient by hand for every architecture. Backpropagation is the calculus you already know, organised so a machine can do that derivation for you, automatically, every time.

The forward pass, precisely

Fix a network with L layers. Layer l takes the previous layer's activations a^(l−1), applies a learned affine map, and passes the result through a nonlinearity g:

The input is a⁽⁰⁾ = x. The final layer's activations a^(L) are the network's output, and a loss L compares that output against the true label — squared error for regression, cross-entropy for classification, the choice does not matter for what follows. Every quantity in this lecture is defined for one training example at a time; summing or averaging over the training set happens outside the derivation, the same way it did for every gradient earlier in this track.

What the backward pass has to compute

Training the network means adjusting every weight matrix W^(l) to reduce L, which requires ∂L/∂W^(l) for every layer l. Take the concrete case of a 3-layer network — layers 1, 2 and 3, with layer 3 the output layer — and ask for ∂L/∂W⁽²⁾, the gradient with respect to the middle layer's weights. W⁽²⁾ influences L only through the path it lies on: W⁽²⁾ sets z⁽²⁾, which sets a⁽²⁾, which (through W⁽³⁾) sets z⁽³⁾, which sets a⁽³⁾, which the loss reads directly. The chain rule says the total derivative along that path is a product of the local derivatives at each link:

Each factor is an ordinary derivative you already know how to write down, and none of them is mysterious once named individually:

• ∂L/∂a⁽³⁾ — how sensitive the loss is to the network's final output. For squared error this is just twice the residual; the specific form depends on the loss, not on anything about the network's depth.
• ∂a⁽³⁾/∂z⁽³⁾ — the output nonlinearity's local derivative, g′(z⁽³⁾), evaluated at the value the forward pass already computed.
• ∂z⁽³⁾/∂a⁽²⁾ — how the pre-activation at layer 3 depends on the activations feeding it; since z⁽³⁾ = W⁽³⁾a⁽²⁾ + b⁽³⁾, this factor is exactly the weight matrix W⁽³⁾ — the same matrix used forward, now carrying the gradient backward.
• ∂a⁽²⁾/∂z⁽²⁾ — layer 2's own nonlinearity's local derivative, g′(z⁽²⁾).
• ∂z⁽²⁾/∂W⁽²⁾ — since z⁽²⁾ = W⁽²⁾a⁽¹⁾ + b⁽²⁾, this factor is simply a⁽¹⁾, the activation this weight matrix was applied to on the way forward.

Nothing here is asserted; every factor is an ordinary partial derivative of one of the two equations in §14.1, evaluated at the values the forward pass already produced. The chain rule is not a metaphor for backpropagation — this product is backpropagation, written out in full for one weight matrix.

Naming the reusable piece: the error at a layer

Group the first two factors of the product above — the derivatives that depend only on what happens at or after layer 3 — into a single quantity, conventionally written δ⁽³⁾ = ∂L/∂a⁽³⁾ · ∂a⁽³⁾/∂z⁽³⁾, sometimes called the error at layer 3. Once δ⁽³⁾ is known, the same quantity — not recomputed, simply reused — is exactly what is needed to start the computation of δ⁽²⁾:

This is the entire algorithm, stated in general form: compute δ at the output layer directly from the loss, then at every earlier layer obtain δ^(l) from δ^(l+1) — the layer just after it — by one matrix multiply and one elementwise product with the local nonlinearity's derivative. Every layer's weight gradient then falls out as δ^(l) times the previous layer's activation, exactly as the last factor pair of the chain-rule product in §14.2 showed for W⁽²⁾ specifically.

Why this is cheap: reuse, not repetition

The efficiency claim in the algorithm's name is precise and worth stating without hand-waving. If ∂L/∂W⁽¹⁾ were derived completely independently from ∂L/∂W⁽²⁾ — starting over from the loss each time, expanding the full chain rule product from scratch for every layer — the factors belonging to every layer after the one being differentiated would be recomputed again and again, once per weight matrix, in a network with L layers. Computing δ^(l) for every layer in a single backward sweep, starting at the output and working toward the input, means each layer's error is computed exactly once and handed to the layer before it as a ready-made partial result. That reuse — not any deeper mathematical trick — is why the cost of computing every gradient in the network is a small constant multiple of the cost of one forward pass, rather than growing with the number of layers the way independent re-derivation would.

What this buys the field

Every hypothesis class this track has studied through Volume VI — linear and logistic regression, GDA and Naive Bayes, the SVM, a single decision tree — was simple enough that its gradient (or its closed-form solution) could be derived by hand, once, and written down for good. A neural network of arbitrary depth and arbitrary layer composition has no such luxury: nobody is deriving ∂L/∂W^(l) by hand for every new architecture a researcher dreams up. Backpropagation is the general, mechanical procedure — the chain rule applied systematically, in the output-to-input order §14.3 and §14.4 justify — that computes that gradient automatically, for any composition of differentiable layers. That generality, not any single equation in this lecture, is what made deep, many-layered networks trainable at all, including every architecture the sibling track on modern large language models studies.