Volume V · Thinking Before Acting

Reason, Act, Observe, Reflect

Lecture 11 gave reasoning a shape — a chain, or a searchable tree — but neither CoT nor ToT ever touches the outside world; both just think harder about a question already fully specified in the prompt. This lecture is about the step where reasoning starts reaching outside the model: querying something, observing what comes back, and folding that observation into the next thought. Three papers answer "how does an agent get better at using tools" on three genuinely different axes, and this lecture is precise about which axis is which.

ReAct: one stream, three phases, no fine-tuning

ReAct (Yao et al., arXiv:2210.03629, 2022) interleaves free-form reasoning traces — labeled "Thought:" — with task-specific actions — labeled "Act:" — and the environment's resulting observations — labeled "Obs:" — inside a single generation stream. The reasoning traces let the model induce, track, and update its own plan and handle exceptions as they come up; the actions let it reach outside its own context to query an external source, such as a Wikipedia API, mid-episode. Like CoT before it, ReAct is implemented purely via few-shot prompting — no fine-tuning of the underlying model is required to get this behavior.

The paper's reported results split cleanly into two regimes. On knowledge-intensive QA tasks — HotpotQA and FEVER — ReAct is competitive with or beats plain CoT, because grounding reasoning in retrieved observations helps more than reasoning alone. On interactive decision-making tasks — ALFWorld and WebShop — ReAct beats imitation- learning and reinforcement-learning baselines using only 1–2 in-context examples, which is a striking result given those baselines were trained specifically for their environments.

Reflexion: memory across attempts, still no weight updates

Reflexion (Shinn et al., arXiv:2303.11366, NeurIPS 2023) answers a different question: given that an agent just failed at something, how does it get better on the next try without retraining? The paper's own framing is "verbal reinforcement learning" — instead of updating model weights via gradient-based RL, the agent generates a natural-language self-reflection on the feedback signal from a failed trial (which may be a scalar score or free-form text), and stores that reflective text in an episodic memory buffer. On the next attempt, that stored reflection is fed back into the model's context alongside the task. No weight updates happen anywhere in this process — the "learning" is entirely textual and in-context. The approach is flexible to different feedback types (scalar or free-text) and different feedback sources (an external environment or an internally simulated one).

Toolformer: the one that actually retrains the model

Toolformer (Schick et al., arXiv:2302.04761, 2023) is a genuinely different kind of contribution from the two above — a training-time one. The model is fine-tuned, in a self-supervised way, to decide which API to call, when to call it, what arguments to pass, and how to weave the returned result back into its own subsequent token prediction. The training data for this fine-tuning is bootstrapped by the base model itself: candidate API calls are sampled from the model, and only the calls whose downstream loss actually improves are kept as training examples — requiring only a handful of human demonstrations per tool to get the bootstrap started. The tools covered in the paper are a calculator, a question-answering system, two search engines, a translation system, and a calendar.