The Great Mental Models of Machine Learning 20 lectures

A lecture series  ·  Revised July 2026  ·  In the tradition of Stanford CS229

The Great Mental Models of Machine Learning

Every other series on this site studies a specific model. This one studies the question underneath all of them: given a finite sample and an infinite space of possible explanations, what can you actually prove about the one you fit? Full derivations, not just mechanisms — θ, hθ(x), and J(θ) throughout, in the notation the field actually uses.

Twenty lectures · Ten volumes · Full derivations, primary sources throughout

Scroll slowly ↓

A model that perfectly explains the data you have and a model that generalizes to data you don't are not the same object — and the entire discipline of learning theory exists because that gap is measurable before you ever see new data.
Linear and logistic regression, SVMs, decision trees, and neural networks look like different tools. They are the same wager made four different ways: assume a hypothesis class, fit it by an optimization, and pay a provable price in generalization for however expressive that class was allowed to be. This series makes that wager explicit every time, with the maths worked out in full.
I

The generalization problem

A model can memorize its training set perfectly and still fail on the next example. Every regularizer and every learning-theory bound is a response to this one fact.

II

The optimization problem

Most of the objectives worth minimizing have no closed form. Gradient descent, Newton's method, coordinate ascent, and EM are four different ways of climbing a hill you can't see the top of.

III

The labels problem

Labeled data is expensive and sometimes doesn't exist. Generative models, clustering, and dimensionality reduction are what you reach for when supervision runs out.

Volume I

The Linear Story

fitting a line, then bending it, then turning it into a boundary
1 The Line That Learns Itself Least mean squares by gradient descent, the normal equations in closed form, and the probabilistic argument for why squared error was the right loss all along. Linear regression 2 Bending the Line Just Enough Locally weighted regression trades a global assumption for a local one — and previews the bias a model pays for the shape you force on it. LWR · bias 3 From a Line to a Boundary The sigmoid turns a linear score into a probability. Maximum likelihood gives the objective; Newton's method gets there in a handful of steps. Logistic regression 4 One Framework, Many Regressions Linear and logistic regression are both special cases of one exponential-family template. Softmax regression is what the template gives you for free at k classes. GLMs
Volume II

Generative Learning

model the classes, not the boundary between them
5 Learning the Classes, Not the Boundary Gaussian Discriminant Analysis models how each class generates its data and derives the boundary as a consequence — the generative-vs-discriminative split, made concrete. GDA 6 What Independence Buys You Naive Bayes assumes features are conditionally independent given the class — obviously false, and startlingly effective anyway. Laplace smoothing fixes its one real failure mode. Naive Bayes
Volume III

Margins and Kernels

the widest possible street between two classes
7 The Widest Street Between Two Classes The optimal margin classifier as a constrained optimization problem — and why maximizing the margin, not just finding any separating boundary, is what generalizes. SVM 8 The Trick That Isn't Really a Trick The Lagrangian dual exposes the SVM as a function of inner products alone — replace the inner product and you've silently mapped into a higher-dimensional space. Soft margins and SMO make it practical. Kernels · SMO
Volume IV

What Learning Theory Actually Guarantees

provable bounds before you see the test set
9 The Bias-Variance Bargain Every model's test error decomposes into a piece from being too simple and a piece from being too flexible. Hoeffding's inequality is where that decomposition gets a number attached. Bias-variance 10 Counting What a Model Can Fit VC dimension measures a hypothesis class's capacity directly — not by parameter count, but by the largest set of points it can label every possible way. VC dimension
Volume V

Regularizing and Choosing

paying the generalization price on purpose
11 Penalizing Complexity on Purpose L1 and L2 regularization are what a Laplace prior and a Gaussian prior on the weights look like after Bayes' rule. Cross-validation is how you pick how hard to penalize. L1 · L2 · CV
Volume VI

Trees and Committees

splitting, then voting
12 Splitting on What Matters Most A decision tree is a sequence of greedy questions, each chosen to reduce impurity the most. Simple, interpretable, and prone to memorizing the training set. Decision trees 13 Why a Crowd Beats an Expert Bagging averages away variance across resampled trees. Boosting does the opposite — it chains weak learners, each fixing the last one's mistakes. Bagging · Boosting
Volume VII

Neural Networks From First Principles

the chain rule, applied systematically
14 The Chain Rule, Applied Systematically Backpropagation has no separate idea beyond the chain rule — the forward pass computes the function, the backward pass computes its gradient, layer by layer, in reverse. Backprop
Volume VIII

Learning Without Labels

when nobody tells you the groups
15 Grouping Without Being Told the Groups K-means alternates between assigning points to the nearest center and moving each center to its assigned points' mean — provably converges, provably not to a global optimum. K-means 16 The Algorithm That Guesses Twice The EM algorithm guesses which cluster generated each point, then refits the clusters to those guesses, then repeats — and Jensen's inequality is why this is guaranteed to improve. EM · GMM 17 Throwing Away the Right Dimensions PCA can be derived two ways — maximize the variance you keep, or minimize the reconstruction error you lose — and they land on the exact same eigenvectors. PCA
Volume IX

Learning From Reward

when the only signal is a delayed number
18 A World Where Actions Have Consequences Markov Decision Processes formalize an agent acting over time. The Bellman equation is the recursive fact that makes value iteration and policy iteration possible at all. MDPs 19 Learning the Value of an Action Q-learning learns the Bellman equation's solution directly from experience, without ever building a model of the environment. Policy gradients take the opposite route. Q-learning
Volume X

Practicing the Craft

the lecture every practitioner wishes came first
20 Reading Your Own Learning Curves High bias and high variance look different on a training-vs-validation curve, and they call for opposite fixes. A practical diagnostic, not just a definition. Diagnostics