Volume XI · Proving It Works

Where Context Fails, and What a Harness Buys You

Two questions get run together too often: is a model's raw ability to use its context improving, and is the scaffold around it improving. This lecture keeps them apart, because the research behind each answers a genuinely different thing — one is a hard limitation with no harness fix, the other is a capability trend whose two contributing causes are reported together far more often than they are cleanly separated.

A limitation no scaffold removes: Lost in the Middle

Liu et al.'s "Lost in the Middle" (arXiv:2307.03172) evaluates models on multi-document question answering and key-value retrieval — tasks that require pulling one specific piece of relevant information out of a long input context. The finding is a shape, not a single number: a U-shaped performance curve across the position of the relevant information within the context. Models perform best when the answer sits at the very start or the very end of the context, and performance degrades significantly when it sits in the middle — and that degradation persists even for models explicitly marketed for long-context use. This lecture states the finding qualitatively and deliberately does not attach a specific percentage-point drop to it: the fetched abstract for this lecture did not surface a single quantified figure worth citing as if it were precise, and inventing one would violate the standard this whole track holds itself to.

Nothing in a permission mode, a sandbox, or a subagent — the mechanisms of Lecture 22 and Lecture 21 — touches this. It is a property of how the model itself attends across a long context, not a property of what it is permitted to do or where its exploration is contained. The only lever that helps is upstream of the harness entirely: Lecture 21's context engineering — keeping a context small, or placing the load-bearing information deliberately near the edges rather than trusting the middle — works around the limitation without removing it.

A capability trend, less cleanly decomposed than it sounds

METR's "Measuring AI Ability to Complete Long Tasks" (metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/, full methodology at arXiv:2503.14499) defines a "time horizon" metric: the length of task — measured in how long the task takes a human professional — that a frontier model agent can complete autonomously with 50% reliability. That threshold is located by fitting success probability against human task-duration data with a logistic curve and interpolating where the fit crosses 50%. The headline result is a doubling: this time horizon has doubled roughly every seven months across the trailing six years, 2019 through 2025, with METR separately noting acceleration to roughly every four months in the 2024–2025 window.

The caveat matters as much as the headline number. The fetched METR blog post does not itself explicitly decompose "how much of this rise is a better underlying model" from "how much is a better scaffold or harness wrapped around the same model" — it reports time horizon as a single blended metric per model-plus-scaffold combination. Any strong claim that METR "explicitly separates" those two levers should be treated as unconfirmed pending a full read of the methodology paper; the blog post alone does not support it, and press coverage of METR's trend has sometimes implied a cleaner split than the primary source itself makes.

The one place the split actually gets made

The clearest primary statement this lecture found of model quality and scaffold quality treated as genuinely separate, independently measurable levers comes not from METR but from Anthropic's own multi-agent research system writeup, already cited in Lecture 21: "Upgrading to Claude Sonnet 4 yields larger performance gains than doubling the token budget on Sonnet 3.7." That single sentence disentangles the two variables press coverage tends to blur — a better model, on the one hand, and more compute or a bigger scaffold thrown at an older model, on the other — and states directly that the two are not interchangeable, nor equally impactful per unit of resource spent.