Volume IX · Learning From Reward

A World Where Actions Have Consequences

Every algorithm so far in this course has learned from a fixed dataset, handed over all at once, with no memory of how it was collected. This lecture's subject lives somewhere else: a world where its own choices change what happens next, and the only signal telling it how it is doing is a number, and often that number arrives late.

The mental model

A Markov Decision Process names a world in which acting changes what state you are in. The Bellman equation is the single recursive fact — the value of being somewhere is the immediate reward plus the discounted value of wherever you end up next — that makes it possible to compute an optimal way to act at all.

The tuple

Formalise the setting as a Markov Decision Process, a tuple (S, A, {P_sa}, γ, R). S is the set of possible states the world can be in. A is the set of actions available to the agent. {P_sa} is a collection of state-transition probabilities: for each state s and action a, P_sa is a distribution over which state the world moves to next. γ ∈ [0, 1) is a discount factor, controlling how much a reward received later is worth relative to one received now. R is a reward function, giving a number for being in (or, in some formulations, transitioning into) a given state.

"Markov" names the assumption doing the real work here: the transition probabilities depend only on the current state and action, never on the history of how the agent arrived there. Everything relevant about the past is assumed to be summarised in the present state.

Figure 18a A minimal MDP: circles are states, labelled arrows are actions together with the transition probability P_sa(s′) of landing in the state the arrow points to. From s₁, taking action a₁ lands in s₂ seventy per cent of the time and s₃ the rest.

The value of a policy

A policy π is a function mapping states to actions — a complete rule for what to do wherever you find yourself. Fix one policy π and ask a well-defined question: starting from state s and always following π from then on, what is the expected total discounted reward? Call that quantity V^π(s), and it satisfies a recursive identity known as the Bellman equation:

Bellman equation, for a fixed policy π V^π(s) = R(s) + γ Σ_s′ P_s,π(s)(s′) V^π(s′)

Read this equation exactly as it is written. The value of being in state s under policy π is the immediate reward R(s) collected just for being there, plus γ times the expected value of wherever the world sends you next — averaged over every possible next state s′, weighted by how likely π's chosen action is to land you there — and that next value, V^π(s′), is defined by the very same equation, one step further out. The recursion is exact, not an approximation: it holds because the discounted future reward from s decomposes cleanly into "the reward right now" plus "everything after, discounted by one factor of γ."

The best possible policy

Rather than fixing a policy and asking its value, ask directly: what is the best achievable value at every state, over every possible policy? Call that V*(s), and it obeys the same recursive structure with one change — instead of following a fixed action π(s), take the maximum over every action available:

Bellman optimality equation V*(s) = R(s) + max_a γ Σ_s′ P_sa(s′) V*(s′)

Once V* is known, the optimal policy at every state falls out immediately: at state s, take whichever action a maximises the term being maximised above. The entire planning problem — what should the agent do, everywhere, forever — reduces to computing a single function V* over the state space.

Two ways to compute it

Value iteration computes V* directly, by turning the Bellman optimality equation into an update rule and applying it repeatedly to every state at once:

Value iteration, the update for every s: V(s) := R(s) + max_a γ Σ_s′ P_sa(s′) V(s′)

Repeat this update over all states until V stops changing appreciably. At the mechanism level, the reason this is guaranteed to work — rather than merely a plausible heuristic — is that the update is a contraction mapping: each application strictly shrinks the largest possible gap between the current V and the true V*, and a contraction applied repeatedly is guaranteed to converge to its unique fixed point. The full convergence proof belongs to the theory of contraction mappings and is cited here rather than re-derived; the fixed point it converges to is exactly V*.

Policy iteration takes a different route to the same destination, alternating two steps. Policy evaluation fixes the current policy π and solves the linear system given by its Bellman equation exactly, obtaining V^π in full. Policy improvement then updates the policy greedily at every state: choose whichever action looks best according to the just-computed V^π. Alternate evaluation and improvement, and repeat until the policy itself stops changing between rounds.

Figure 18b The values inside two states, updated round by round by the Bellman update, settling toward stable numbers — value iteration made visible. The gap between successive rounds shrinks each time, which is the contraction property doing its work.

Plate 18.1 Step through the Bellman update round by round on the lecture's own four-state MDP and watch V(s) converge live — reproducing the static figure's 0.0 → 1.0 → 2.7 progression exactly — then drag γ down and watch the same contraction settle faster, onto a lower ceiling.

What both algorithms assume

Name the assumption both of these algorithms share, because the next lecture exists entirely to remove it. Value iteration and policy iteration both require the transition probabilities P_sa and the reward function R to be known in advance — a complete model of how the world responds to every action, before a single action is ever taken. That is a strong assumption: a robot navigating an unfamiliar building, a game whose rules are unknown, or a market whose dynamics are not written down anywhere does not hand you P_sa for free.

The next lecture asks the question this one leaves open: what do you do when you do not have a model of the world at all, and can only learn from lived experience — the actions actually taken, the rewards actually received, and nothing more?

Read the primary source

CS229 Lecture Notes — Reinforcement Learning and Control — Stanford. MDPs, the Bellman equation, value iteration and policy iteration, derived in full.
CS229 · Reinforcement Learning — Aman.ai. Cross-verified transcription of the same material, useful as a second pass.

❦