Volume VIII · Measuring an Agent
Four Benchmarks, Four Kinds of Task
A demo shows an agent succeeding once, on a task someone chose. A benchmark is the opposite bet: hundreds of tasks nobody cherry-picked, scored the same way every time, so that a number can be compared across models and across years. The four benchmarks in this lecture were each built to catch a different way an agent fails, and each was published with the state-of-the-art model of its day scoring far below a human doing the same task.
The mental model
SWE-bench, GAIA, WebArena, and AgentBench each isolate a different axis of "can an agent actually do the job" — real code repair, verifiable multi-step tool use, functional web interaction, and breadth across many environments. Read what a benchmark's tasks actually require before trusting its leaderboard number; the number without the task design underneath it tells you almost nothing.
SWE-bench: a patch that has to actually resolve the issue
SWE-bench (Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", arXiv:2310.06770, ICLR 2024) is built from 2,294 real issue/pull-request pairs pulled from 12 popular Python repositories. Given a snapshot of the codebase and the text of a real reported issue, the model has to produce a patch that resolves it — and because these are real issues from real repositories, a correct patch routinely requires coordinated edits across multiple files and multiple functions, not a single self-contained fix. The task is scored by running the repository's own test suite against the produced patch: either the tests the maintainers wrote to confirm the fix now pass, or they don't.
The paper's own baseline finding, at 2023 publication, was blunt: the models it tested could resolve only the simplest subset of the 2,294 issues — the kind requiring a small, localized change, not the multi-file coordination that makes up a meaningful share of the benchmark. SWE-bench's contribution was not a clever task format; it was refusing to synthesize the tasks at all, and inheriting whatever difficulty real software maintenance actually has.
GAIA: an unambiguous string at the end of a messy process
GAIA (Mialon et al., "GAIA: A Benchmark for General AI Assistants", arXiv:2311.12983) takes the opposite design decision on scoring. Its 466 questions demand reasoning, handling of multiple modalities, web browsing, and tool use in whatever combination the question requires — but every answer is a short, unambiguous factual string, chosen specifically so the benchmark can be scored automatically rather than needing a human grader to judge a long free-text response. The difficulty lives entirely in the path to the answer, not in interpreting what counts as correct once you have it.
GAIA's disclosed numbers make the gap stark: human annotators scored 92% accuracy on these questions. GPT-4 augmented with plugins — the strongest configuration tested at the paper's 2023 publication — scored 15%. A benchmark engineered to be trivial to grade turned out to be far from trivial to solve.
WebArena: correctness means the website actually changed
WebArena (Zhou et al., "WebArena: A Realistic Web Environment for Building Autonomous Agents", arXiv:2307.13854) scores a different kind of correctness again. Its 812 long-horizon tasks run across four realistic, fully-functional website domains — e-commerce, forum discussion, software development/collaboration, and content management — and a task is scored on whether the underlying application state actually ends up correct: an order actually placed, a post actually edited, a ticket actually closed. Producing text that describes the right action is not the same as the environment reflecting it, and WebArena's scoring is built to tell the two apart.
The disclosed gap here is the widest of the four: the best GPT-4-based agent tested at publication (2023) achieved a 14.41% success rate, against a human success rate of 78.24% on the identical task set. Long-horizon, multi-step web interaction — clicking through a real interface toward a real state change — was, at that point, close to unsolved.
AgentBench: breadth reveals a different gap entirely
AgentBench (Liu et al., "AgentBench: Evaluating LLMs as Agents", arXiv:2308.03688, ICLR 2024) does not chase one hard task type — it spreads across eight distinct environments: operating system interaction, database operations, knowledge graph querying, a digital card game, lateral thinking puzzles, ALFWorld's simulated household tasks, WebShop's shopping environment, and Mind2Web-style browsing. The point of testing across eight environments at once, with 27 different API-based and open-source models, was not to produce one difficulty number but to reveal where capability clusters.
What it revealed was a gap of a different shape than the other three benchmarks: a significant capability separation between top commercial models and open-source models, with the paper citing poor long-horizon reasoning, weak decision-making, and unreliable instruction-following in the open-source models as the main bottlenecks. AgentBench's finding is not "agents are far below humans" — it is "agents are far from uniform across models," which is a distinct and equally load-bearing fact for anyone choosing which model to build an agent on.
| Benchmark | What it measures | Task count | Human vs. model gap (disclosed) |
|---|---|---|---|
| SWE-bench | Real GitHub issue → resolving code patch | 2,294 | No single top-line %; 2023 baselines resolved only the simplest subset |
| GAIA | Multi-step, tool-using reasoning to a verifiable answer | 466 | Human 92% vs. GPT-4 + plugins 15% |
| WebArena | Functional task completion on real websites | 812 | Human 78.24% vs. best GPT-4 agent 14.41% |
| AgentBench | Breadth across 8 distinct environments | 8 environments, 27 models tested | No human baseline disclosed; large commercial-vs-open-source model gap |
Read the task design before the leaderboard number
Every one of these four benchmarks was published with its best contemporary model scoring far below human level, on tasks the authors deliberately built to resist shortcuts — real repositories instead of synthetic bugs, exact-match strings instead of graded prose, live application state instead of a transcript, many environments instead of one favorable one. The discipline this lecture is teaching is to ask what a benchmark's tasks actually require and how they're actually scored before treating its number as comparable to any other benchmark's number, or to a demo at all.
Read the primary source
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, Jimenez et al., ICLR 2024.
- GAIA: A Benchmark for General AI Assistants, Mialon et al.
- WebArena: A Realistic Web Environment for Building Autonomous Agents, Zhou et al.
- AgentBench: Evaluating LLMs as Agents, Liu et al., ICLR 2024.