Volume VIII · Measuring an Agent

Coding Agents, Claims and Scrutiny

Three products claimed to be a coding agent within roughly a year of each other, and they made three very different kinds of claim. One led with a headline percentage that later drew public scrutiny of its demos. One led with almost no autonomy claim at all, engineering every stage to be human-edited before it runs. One led with neither a benchmark number nor an autonomy pitch, but with a documented sandbox. This lecture reads all three the way Lecture 17 asked you to read a benchmark — by what is actually documented, not by what is asserted.

Devin: a strong number, and a comparison basis worth reading closely

Cognition Labs' own blog post, "Introducing Devin," states a 13.86% resolve rate on SWE-bench end-to-end, compared against a stated prior state-of-the-art of 1.96%. Read that comparison carefully before repeating it: both numbers come from Cognition's own blog post, not from an independently run or peer-reviewed leaderboard entry. That does not make the 13.86% figure false — but it means the comparison basis, the exact evaluation setup, and the identity of whatever "1.96%" configuration is being compared against were not independently verified by a third party at time of publication. A number a company publishes about its own product, compared against a number the same post also supplies, is a self-reported claim in both halves.

Separately — and this needs to be flagged in exactly these terms, not folded into the paragraph above as if it were the same kind of fact — multiple independent developers and outlets reported, after Devin's public demo videos circulated, that those demos appeared heavily curated and edited: hand-picked tasks, compressed timelines, and real- world usage reports describing something closer to a supervised junior engineer working under close guidance than an autonomous senior one working alone. This is unconfirmed, press-sourced commentary under scrutiny — disputed marketing claims, not verified mechanism — and it belongs in this lecture precisely because it is the cautionary case: a strong self-reported number followed by public scrutiny of what actually produced it.

Copilot Workspace: the deliberately conservative counter-example

GitHub Copilot Workspace (documented at github.blog/news-insights/product-news/github-copilot-workspace, GitHub's own product blog and docs) makes almost the opposite bet. Its flow is anchored to a specific GitHub issue or pull request, and it proceeds in three explicit stages: first an editable specification of what the change should do, then an editable step-by-step plan for how to do it, then the generated code changes themselves — with every stage, spec, plan, and code, open for the developer to edit before it is accepted and moved forward. It runs on GitHub Codespaces, giving it an in-browser file explorer and integrated terminal, and a workspace session can be shared by link for collaborative review before anything merges.

GitHub's own documentation is explicit about the positioning: this is a human-in-the- loop tool, not an autonomous one, stressing "you, the developer, in control" at each of the three stages. Where Devin's public narrative centered a resolve-rate percentage, Copilot Workspace's public narrative centers the opposite thing — how much of the process a human gets to see and change before it executes.

Codex CLI: leading with the sandbox, not the number

OpenAI Codex CLI (documented at developers.openai.com/codex) is a third kind of claim again — its public documentation leads with architecture and safety mechanics rather than a headline benchmark. The implementation is described as a thin TypeScript launcher over a Rust core; the Rust core covering the CLI/TUI, the agent's core loop, sandboxing, authentication, and MCP support. That "Rust core, 60-plus crates" level of detail is worth flagging specifically: it comes from a third-party technical wiki, not confirmed directly from OpenAI's own documentation in this research pass, and should be treated as secondary and unconfirmed even though the broader TypeScript-launcher- over-Rust-core split is documented.

The sandboxing itself is documented directly. Codex CLI uses a defense-in-depth approach built on platform-native OS security primitives; network access is disabled by default during command execution, and file operations are scoped to the current directory tree by default. On top of that, the docs describe three approval modes, with "Suggest" — the agent proposes every file edit and every command, and nothing executes without explicit user approval — as the default and safest starting mode.

Line these three up and the lesson of this lecture is not "Devin is bad and the others are good" — it is that a benchmark percentage, a human-in-the-loop workflow, and a sandboxing default are three different kinds of evidence, earned in three different ways, and a reader has to know which kind they are looking at before deciding how much to trust it. That is exactly the discipline the next volume needs: from here forward, this series stops asking what an agent can do, and starts asking how the scaffold around it is built — sandboxing, approval modes, and editable plans are the first specimens of that scaffold, not incidental features of these three products.