Volume IX · Learning From Reward

Learning the Value of an Action

Value iteration and policy iteration both need a map of the world drawn before they will plan a single step inside it. Most of the settings worth acting in do not come with that map. This lecture's algorithm throws the map away entirely and learns the worth of an action the only way it can — by taking it, and watching what happens.

A function of actions, not just states

The previous lecture's V(s) answers "how good is it to be in state s", but acting on that answer still requires knowing which action gets you to the best next state — which is precisely the P_sa that is being assumed away here. Q-learning sidesteps this by learning a different function altogether: Q(s, a), the expected discounted return of taking action a in state s right now, and then acting optimally forever after.

Once Q is learned, choosing what to do at any state s no longer needs a model of the world at all — it needs only a maximum over a finite list of numbers:

That single line is the entire payoff of learning Q instead of V. Crucially, Q is itself learned purely from observed transitions the agent actually experiences — a state, an action taken, the reward received, and the state landed in — updated incrementally as those transitions occur, with no transition probabilities P_sa ever written down or estimated at any stage.

This is precisely what makes Q-learning model-free — a term worth naming explicitly, in direct contrast to the previous lecture. Value iteration and policy iteration are model-based: both require P_sa and R to be known in advance, before any planning can begin. Q-learning requires neither, at any point in the process. Its origin is Watkins (1989), Learning from Delayed Rewards, a PhD thesis submitted to King's College, University of Cambridge — cited here by its standard bibliographic details, since the specific PDF link for the thesis was located by search but not independently re-fetched and confirmed live this session.

The other family — adjust the policy directly

Q-learning still learns a value-like function, just indexed by (state, action) instead of state alone. A third family skips value functions altogether. Rather than learning what any state or action is worth, directly parameterise a policy π_θ(a | s) — a distribution over actions, given a state, with learnable parameters θ — and adjust θ by gradient ascent on expected reward. This is REINFORCE, due to Williams (1992), Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning, Machine Learning, 8, 229–256, DOI 10.1007/BF00992696.

The gradient identity that makes this practical is cheap enough to state directly even without the full derivation:

Read this the way the rest of the course reads a gradient: it says what direction to nudge θ in. Increase the probability of whichever actions were followed by high reward, and do so in direct proportion to how much reward followed — the log π_θ term supplies the direction, and the reward R supplies how hard to push in that direction.

Three answers, one question

Volumes VIII and IX of this course share a structure worth naming explicitly now that all three answers are on the table. Model-based planning — value iteration and policy iteration, from the previous lecture — needs a known model of the world before it computes anything. Model-free value learning — Q-learning — learns action-values from raw experience with no model at all, but still learns a value-like quantity as an intermediate step. Policy gradients skip value functions altogether, adjusting the policy's own parameters directly from the reward signal.

These are three different answers to the same underlying question: what should be learned when a model of the environment is unavailable, or too expensive to build, or simply beside the point for the decision actually in front of the agent. None of the three is strictly superior — each trades off differently between how much of the environment's structure it assumes and how directly it optimises for the behaviour actually wanted.