The Great Mental Models of AI Systems 25 lectures

A lecture series  ·  Revised July 2026  ·  Infrastructure, agents, and the harness between them

The Great Mental Models of AI Systems

Every other series on this site studies a model. This one studies everything a model sits inside of: the silicon and the network fabric that trains and serves it, the loop of reasoning and tool calls that turns it into an agent, and the software scaffold — permissions, context management, evaluation — that decides whether that agent is safe and effective or neither.

Twenty-five lectures · Eleven volumes · Primary sources throughout — hedged explicitly where a claim is press-sourced, not verified

Scroll slowly ↓

A frontier model and a toy demo can share the same weights and produce wildly different results — because almost none of what separates them lives in the model.
It lives in how many GPUs talk to each other and how fast; in whether the serving stack pages its memory or wastes it; in whether the agent's tools are named well enough for the model to use them correctly; in whether a permission model stops a bad action before it runs. This series is about that surrounding machinery — three layers, one underneath the next: infrastructure, agents, and the harness engineering that ties an agent to its infrastructure safely.
I

The infrastructure problem

A model is only as fast as the hardware training and serving it, and hardware has physical limits — bandwidth, memory, power. Every parallelism strategy and every serving system is a negotiation with those limits.

II

The agency problem

A language model that only predicts the next token is not yet an agent. Reasoning loops, tool calls, memory, and multi-agent coordination are what turn prediction into action — and each adds a new way for that action to go wrong.

III

The trust problem

An agent that can act is an agent that can act badly. Harness engineering — tool design, context management, permissions, sandboxing, evaluation — is the discipline of making that risk bounded and measurable instead of hoped-for.

Volume I

Silicon

the accelerators everything else is built on
1 The Accelerator Zoo NVIDIA's A100 through GB200, Google's TPU v4 through Ironwood, and AMD's MI300X — what FLOPS, memory bandwidth, and NVLink generation actually mean for a training run. GPUs · TPUs 2 Wiring the Datacenter Together NVLink, NVSwitch, and InfiniBand — why a cluster's interconnect matters as much as its chips, with Meta's disclosed Llama 3 cluster as the worked example. Interconnects
Volume II

Splitting the Work

a model too large for one chip, or one machine
3 Cutting a Model Into Pieces Data parallelism replicates the model; tensor parallelism cuts inside a layer. Megatron-LM's approach to splitting attention and MLP blocks across GPUs with minimal communication. Tensor parallelism 4 Stages, Bubbles, and Redundant Memory Pipeline parallelism splits a model by layer instead of within one — and pays for it in idle "bubble" time. ZeRO's three stages remove redundant optimizer-state memory instead. Pipeline · ZeRO 5 Splitting the Sequence, Splitting the Experts Ring Attention shards a single long sequence across devices; expert parallelism shards a mixture-of-experts model by which expert lives where. 3D parallelism composes all of it at once. Sequence · MoE parallelism
Volume III

Serving It Back

a trained model is not yet an answer
6 Paging the KV Cache vLLM's PagedAttention borrows virtual memory's oldest trick — treat the KV cache like paged RAM instead of one contiguous block — and Orca's iteration-level scheduling batches requests that don't even start together. vLLM · continuous batching 7 Guessing Several Tokens at Once Speculative decoding lets a small draft model propose tokens a large model verifies in parallel — provably preserving the large model's exact output distribution. SGLang's RadixAttention reuses KV cache across calls. Speculative decoding 8 Splitting Prefill From Decode Prefill and decode want opposite things from a GPU — compute-bound versus memory-bound. DistServe's answer is to stop making one pool of GPUs do both. Disaggregated serving
Volume IV

The Reality of Scale

what the papers admit actually happens
9 The Utilization Nobody Reaches 100% Of Model FLOPs Utilization, defined in the PaLM paper, is the honest ratio of useful compute to peak hardware compute. Llama 3's and OPT-175B's disclosed hardware-failure logs show why it is never 100%. MFU · failures 10 Gigawatts, Not Just GPUs Frontier training is now a power-and-money problem before it is a compute problem — Stargate's disclosed capacity commitments, DeepSeek-V3's disclosed training cost, and where reporting outruns disclosure. Power · economics
Volume V

Thinking Before Acting

where an agent's reasoning actually comes from
11 A Chain, Then a Tree Chain-of-thought prompting elicits step-by-step reasoning with no architecture change at all. Tree of Thoughts generalizes the chain into a searchable tree the model can backtrack within. CoT · ToT 12 Reason, Act, Observe, Reflect ReAct interleaves reasoning traces with real actions in one generation stream — the pattern nearly every agent loop still traces back to. Reflexion adds a memory of its own past failures; Toolformer is trained, not prompted, to reach for a tool. ReAct · Reflexion
Volume VI

Reaching Outside the Model

a model that can only talk is not yet an agent
13 The JSON Contract Function calling is a schema, a stop reason, and a result block passed back into the conversation — not magic. What actually happens between a tool_use block and the code that executes it. Tool use 14 From M×N to M+N Model Context Protocol standardizes the interface between AI applications and tools so each side implements it once. Hosts, clients, servers, and the primitives — tools, resources, prompts — that pass between them. MCP
Volume VII

Memory and Many Minds

what an agent remembers, and who it works with
15 RAM and Disk, for a Language Model RAG retrieves external text into context on demand. MemGPT goes further — the model itself pages information between its context window and external storage, the way an OS pages virtual memory. RAG · MemGPT 16 One Agent Becomes Several AutoGPT and BabyAGI's early, brittle loops; AutoGen's conversable-agent framework; and Anthropic's own disclosed orchestrator-worker research system, with its real token-cost multiplier. Multi-agent systems
Volume VIII

Measuring an Agent

a demo is not a benchmark
17 Four Benchmarks, Four Kinds of Task SWE-bench's real GitHub issues, GAIA's tool-using reasoning questions, WebArena's functional websites, and AgentBench's eight environments — what each actually measures, and the gap between humans and models on every one. SWE-bench · GAIA 18 Coding Agents, Claims and Scrutiny Devin's self-reported number against the scrutiny its demos received; GitHub Copilot Workspace's deliberately human-in-the-loop design; Codex CLI's sandboxing defaults — separating documented mechanism from marketing. Devin · Copilot · Codex
Volume IX

Designing the Tools

a tool is a contract with something that cannot read your mind
19 Workflows, Agents, and When to Use Which Anthropic's own taxonomy: workflows follow predefined code paths, agents direct themselves. Five named patterns, each with a stated fit condition — and the standing advice to add agentic complexity only when a simpler system falls short. Agent design patterns 20 Writing a Tool a Model Won't Misuse Description quality, error messages that teach instead of just failing, consolidating ten thin tools into one, and an eval-driven refinement loop that measurably beat human-written tool definitions. Tool design
Volume X

Managing Context and Trust

what an agent is allowed to remember, and to do
21 Caching a Context, Isolating a Subagent Prompt caching's breakpoints and TTLs turn a repeated long context into a cost and latency win. Subagent isolation keeps a main conversation's context clean — at a disclosed token-cost multiplier. Context engineering 22 What an Agent Is Allowed to Touch Claude Code's six permission modes, OS-level sandboxing as a second, independent layer, and the "lethal trifecta" — private data, untrusted content, and an exfiltration channel — that makes prompt injection dangerous. Permissions · sandboxing
Volume XI

Proving It Works

the synthesis — where infrastructure, agency, and trust meet
23 Testing an Agent Like a System, Not a Chatbot SWE-bench Verified's human-filtering process and Terminal-Bench's outcome-only Docker tasks — sandboxed, deterministic, pass@k, never trusting the agent's own claim of success. Evaluation harnesses 24 Where Context Fails, and What a Harness Buys You Lost in the Middle's U-shaped retrieval curve, the lethal trifecta's exfiltration risk, and METR's time-horizon-doubling metric — with Anthropic's own finding that a better model and a bigger scaffold are separate, comparable levers. Failure modes · METR 25 Reading an Agent System End to End The synthesis: how a real 2026 coding agent composes all three layers of this series — the cluster underneath, the reasoning loop in the middle, and the harness holding the risk at every layer to something bounded. Synthesis