Volume V · Reading the Reports

Judging What a Model Sees

A model can answer ninety per cent of a benchmark's questions correctly and still describe a dog sitting on a couch that has no dog on it at all, if you simply ask "is there a dog here" in the right way. A benchmark score and a model's actual reliability are two different claims, and reading a report well means never quietly treating one as the other.

A benchmark built to resist captioning

MMMU — A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI, published in November 2023 — was built specifically because the earlier generation of vision-language benchmarks could be beaten by fluent picture-captioning without any real expert reasoning underneath it. Its construction is worth reporting precisely, because the numbers are the argument: 11,500 multimodal questions, drawn from real college-level exams, quizzes, and textbooks, spanning six broad disciplines — Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering — across 30 subjects and 183 subfields, and drawing on 30 heterogeneous image types that include charts, diagrams, maps, music sheets, and chemical structures.

None of that is decoration. A benchmark drawn from real exam material, spanning chemical structures and music sheets in the same test, cannot be passed by a model that has merely learned to describe photographs well — it demands genuine college-level subject knowledge combined with visual reasoning over image types that a captioning model has no reason to have ever seen paired with expert-level questions.

A low score as evidence the design worked

The headline result at publication is worth stating exactly, because it is the clearest evidence that the benchmark achieved its own design goal: GPT-4V, the strongest vision-language model available at the time, scored only 56% on MMMU. A benchmark that a frontier model clears at just above half marks is doing its job — it was built to be hard in a way that simple picture description cannot fake its way through.

MMMU-Pro followed as a more robust, more adversarial version of the same benchmark, built specifically to resist the ways models had already started gaming the original — a pattern that recurs across every benchmark this lecture covers, and worth watching for whenever a report cites a benchmark score without also naming which version of it.

Turning hallucination into a yes-or-no question

Object hallucination — a model confidently describing something in an image that is not there — is hard to measure from open-ended captions, because judging a caption's accuracy is itself a subjective human task with no stable ground truth to grade against. POPE — Evaluating Object Hallucination in Large Vision-Language Models — converts the whole problem into something far more measurable: a binary polling task. Rather than asking a model to caption an image and then arguing over whether the caption hallucinated, POPE simply asks, repeatedly, "Is there a {object} in the image?" and scores the yes-or-no answer against ground truth.

What makes POPE more than a simple spot-check is how it chooses the negative examples — the objects it asks about that are genuinely absent from the image. It uses three distinct negative-sampling strategies: random objects unrelated to the scene, popular objects that appear often across the dataset generally, and adversarial objects chosen specifically because they co-occur frequently with what actually is in the image, making them the hardest wrong answer to correctly reject.

It would be a mistake to treat POPE as a closed, solved measurement. Multiple 2024 and 2025 follow-ups extend the same polling idea beyond simple object presence — to attributes, counting, position, and colour. That proliferation of extensions is itself evidence worth taking seriously: hallucination and grounding remain active, unsolved measurement problems, not something a single benchmark from 2023 closed off for good.

What a score answers, and what it doesn't

Put MMMU and POPE side by side and the teaching point of this lecture becomes concrete rather than abstract. MMMU asks: given a genuinely hard, cross-discipline question, how often does the model reason its way to the correct answer. POPE asks: given a simple, repeatable yes-or-no question about what is actually in an image, how often does the model answer honestly instead of hallucinating. Both are real, falsifiable, valuable measurements. Neither one is the same claim as "this model can be trusted," and a report that leads with a single MMMU number while saying nothing about hallucination rate has answered only one of the two questions a reader actually needs answered.

The discipline this lecture asks you to carry forward is simple to state and easy to forget under the pressure of a clean headline percentage: read a benchmark score as the answer to the specific, narrow question that benchmark was built to ask, and go looking separately for whatever question it was not built to ask — because the gap between the two is exactly where a report's silence tends to live.