Volume XI · Proving It Works
Testing an Agent Like a System, Not a Chatbot
Volume XI is a coda, and this lecture opens it with the question every prior lecture in this series has been building toward without asking directly: once an agent can act, how do you actually know it acted correctly? Not "did it produce a plausible-looking transcript" — did it change the world the way it claimed to.
The mental model
An agent's own narration of what it did is not evidence that it did it. A benchmark that trusts a self-report is measuring confidence, not correctness. The two evaluation designs in this lecture both refuse that trust by the same method: run the agent in a disposable, isolated environment, then check the resulting state with a test the agent did not write and cannot see in advance.
Auditing the benchmark itself: SWE-bench Verified
SWE-bench, before this lecture's subject existed, already asked agents to resolve real GitHub issues against real repositories — Lecture 17 covered it as one of four benchmark designs. SWE-bench Verified (openai.com/index/introducing-swe-bench-verified/) is OpenAI's human-filtered subset of that same test set, built together with the original SWE-bench authors: 500 instances kept out of the full pool, after professional software developers reviewed each candidate. The reviewing process itself is the part worth dwelling on — each problem was reviewed by three experts independently, screening specifically for correctly-scoped unit tests, issue descriptions that are actually solvable from the information given, and the removal of instances carrying grading bugs or test patches so narrow they unfairly penalized otherwise-correct solutions.
The disclosed numbers make the case for why that filtering mattered. GPT-4o resolves 33.2% of SWE-bench Verified with OpenAI's best-performing scaffold, against only 16% on the original, unfiltered SWE-bench. Roughly half of the apparent difficulty in the original benchmark was not the model failing at software engineering — it was the benchmark itself containing unsolvable or mis-scored instances. A model's benchmark score is only as trustworthy as the benchmark's own construction, and constructing one well took the same kind of independent, adversarial checking this lecture argues an agent's own output needs.
One further note belongs here, flagged rather than asserted with confidence: OpenAI has since published a follow-up titled "Why SWE-bench Verified no longer measures frontier coding capabilities." That title is confirmed; its content was not independently fetched for this lecture. Even naming it is a useful data point on its own — a benchmark's own creator judging it saturated or gameable within a couple of years of building it is itself a lesson about how quickly a benchmark can decay from "discriminating" to "solved by everyone who tries."
Outcome over process: Terminal-Bench
Terminal-Bench (tbench.ai/about; github.com/harbor-framework/terminal-bench; arXiv:2601.11868) takes a different shape. Each task pairs an instruction with a Docker image, a set of automated tests, an example solution, and a time limit, spanning scientific computing, software engineering, machine learning, security, system administration, and data science. Its defining property is that it is outcome-driven: the automated tests check the final state of the container against the instruction's stated outcomes, not the method the agent used to get there. An agent is free to solve a task any way it can — the test does not care whether it used a particular library, wrote a particular script, or took a particular sequence of commands, only whether the container ends up in the state the task demanded.
Isolation is structural, not incidental: each task runs in its own Docker container, so one task's side effects cannot contaminate another and the same task can be re-run and still mean the same thing. The v2.1 methodology runs tasks through the "Terminus 2" agent harness inside an e2b sandbox, reporting pass@1 averaged over three repeats per task — repeated sampling rather than a single run, because a single successful attempt does not establish that an agent reliably succeeds.
Stated generally, the pattern both benchmarks share is: a sandboxed, disposable execution environment, deterministic post-hoc test assertions against the final state rather than the process, and repeated sampling reported as pass@1 or pass@k rather than a single trusted run. None of the three pieces is optional — drop the sandboxing and one task contaminates the next; drop the outcome focus and a lucky-looking process gets credited for a result it did not actually produce; drop the repeated sampling and a single success gets mistaken for reliability.
The parallel this track has been living inside
This is the same discipline this teaching site's own build process claims for itself — its house rules state "no proxy verification" outright: a passing test suite the agent wrote itself, a 200 status code, or a non-zero file size is not proof that a change works. A test that failed before a fix and passes after, or an artifact actually opened and checked value by value, is. SWE-bench Verified's human-audited instances and Terminal-Bench's outcome-only container checks are the same idea scaled up to whole benchmark suites: don't trust the thing under test to grade itself, and don't trust a benchmark's own difficulty score until someone has independently checked what it is actually measuring.
Read the primary source
- Introducing SWE-bench Verified — OpenAI.
- Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities — OpenAI.
- Terminal-Bench — About.
- Terminal-Bench — GitHub (Harbor framework).
- arXiv:2601.11868 — Terminal-Bench.