Volume X · Practicing the Craft

Reading Your Own Learning Curves

Every algorithm in this course can fail in exactly two ways: being too simple for the problem in front of it, or being too sensitive to the exact sample it happened to be trained on. A single plot, built from numbers a practitioner already has lying around, tells which failure is actually occurring — before a session is spent guessing at the wrong fix.

The plot Lecture 9 was always pointing at

Lecture 9 introduced bias and variance as a trade a model pays, quantified with Hoeffding's inequality before either quantity is ever measured directly. This lecture asks the practical follow-up: how do you tell which side of that trade a real, already-trained model is actually paying, using only the numbers a training run produces on its own? The instrument is the learning curve — training error and validation error, each plotted against the size of the training set used to produce that point.

Train the same model repeatedly on increasing subsets of the available data — a tenth of it, a fifth, a half, all of it — recording both the training error on whatever subset was used and the error on a fixed held-out validation set each time. Two curves emerge, and the shape formed between them is the entire diagnostic.

Reading the two shapes

The two qualitative shapes are general, well-established diagnostic patterns rather than a specific formula, and each licenses a different response. When training error and validation error are both high and close together, that pattern is generally recognised as high bias: the model's hypothesis class is too simple for the underlying relationship, and it is systematically wrong on the training data itself, not merely on new data. More training data typically will not help here, because the model is not struggling to generalise what it has learned — it is failing to represent the true relationship at all, no matter how much data it is shown. More complexity, or better features, is what the pattern calls for.

When training error is low but validation error is substantially higher, leaving a visible and persistent gap, that pattern is generally recognised as high variance: the model has fit particulars of the training sample that do not generalise. Here more training data typically does help, since a larger sample narrows down the noise the model is over-fitting. Reducing model complexity, or applying the regularisation derived back in Lecture 11, is the other lever available.

A framework beyond the single plot

CS229's public materials document a genuine, confirmed diagnostic framework that extends past the learning curve to systems built of several components chained together. Error analysis asks, for a multi-stage pipeline, which component actually contributes the most error — tested by feeding each stage its ground-truth input in turn and measuring how much the final error improves, which isolates where the error is actually accumulating rather than where it seems to be. Ablative analysis asks the complementary question: remove or disable parts of a working system, one at a time, and measure how much performance drops, which reveals which components actually matter rather than which ones were assumed to matter by design. Mistake analysis asks a third question of a system that is already making errors: categorise the actual mistakes it makes, so that effort goes toward whichever category of mistake is most worth fixing first, rather than the category that happens to be most visible.

It is worth being direct about what is and is not confirmed here. A specific "if high bias then do X, if high variance then do Y" prescriptive checklist is widely taught in introductory machine learning material, and such a checklist exists in ML folklore — but it was not located verbatim in the specific CS229 source fetched this session. The three-part framework above — error analysis, ablative analysis, mistake analysis — is the part directly confirmed in CS229's public materials, and it is presented here on that basis rather than as an unverified checklist dressed up as a citation.

The whole course, retraced

This track's own landing page named three costs at the outset: the generalisation problem, the optimisation problem, and the labels problem. Twenty lectures later, it is worth stating plainly which volumes answered which. Volumes IV and V — learning theory and regularisation — answer the generalisation problem directly, giving an actual number to the price a hypothesis class pays for its own expressiveness. Nearly every derivation across all twenty lectures — gradient descent, Newton's method, the EM algorithm, value iteration — is itself a different answer to the optimisation problem: how to climb toward a good hypothesis when no closed form hands it to you. Volume VIII — k-means, EM, and this volume's PCA — is the course's answer to the labels problem, for the settings where no y is available to check against at all.

That is the note this course ends on. Not a final algorithm, but the discipline of looking at a model's own behaviour before assuming which fix it needs — the same discipline, in miniature, that produced every derivation that came before it.