Volume V · Reading the Reports

Pointing at the Picture

Ask a vision-language model to describe a photograph and it will, in fluent, confident prose, tell you a great deal about what is in it. Ask it to point — to say exactly where, in pixel coordinates, the thing it just described actually sits — and the same fluent model often has nothing to offer. Describing and pointing are different skills, and almost no training data teaches the second one directly.

A gap easy to miss in a benchmark table

The gap this lecture is named for is easy to overlook precisely because it hides inside ordinary-looking evaluation numbers. A benchmark table that reports accuracy on multiple-choice or free-text questions can look uniformly strong even while the model behind it cannot do something as basic as saying, precisely, where in the image the object it just named actually is. A model can score well on "what is in this picture" while having no learned way to answer "where, exactly."

Molmo and PixMo's stated goal

Molmo and PixMo — Ai2's Open Weights and Open Data for State-of-the-Art Vision-Language Models, published in September 2024 — states its own goal precisely, and it is worth reporting in the paper's own terms: prove that a fully open model, meaning open weights, open training data, and an open training recipe together rather than open weights alone, can match closed frontier models, while adding a genuinely new capability most VLMs lack. That last clause is the reason this lecture exists — the new capability in question is pointing.

The data contribution is where the paper's real weight sits, and three pieces of it are worth naming individually. First, dense image captions collected through human speech transcription — a person describing an image out loud, transcribed, rather than a caption distilled from another VLM's outputs, which deliberately avoids inheriting whatever errors that other model had already learned to make. Second, free-form image question-answering data. Third, and the one this lecture is named for: an original 2D pointing dataset, in which the model learns to output actual image coordinates in response to "where is X," rather than settling for a bounding box or a purely textual description of location.

What the result actually claims

The reported result is worth stating exactly as the paper frames it: the 72B Molmo model reportedly beats Claude 3.5 Sonnet and Gemini 1.5 Pro and Flash, and comes second only to GPT-4o, on both academic benchmarks and human evaluation. Read against this lecture's own opening point, that result is notable less for the ranking itself than for what it proves about the ranking's cause: an openly documented model, trained on openly published data with an openly published recipe, reached near-frontier performance while also being the one model in that comparison set that can point.

That combination is the paper's real argument. It is not simply "open models can be competitive" — plenty of open-weights releases had already made some version of that claim. It is that the specific gap this lecture opened with, the one a benchmark table can hide, is closeable with the right data, rather than being some inherent limitation baked into vision-language architecture generally. Molmo's contribution is evidence for that claim, not merely an assertion of it.

Where the lineage goes next, held loosely

Ai2's own blog describes a newer suite referred to as "Molmo 2," which extends pointing to video — both where an object is and when, across a clip's duration — and adds multi-image and multi-video analysis on top of the original single-image pointing capability. This lecture states plainly what it can and cannot confirm about that follow-up: the specific arXiv identifier for Molmo 2 was not independently confirmed this session, so the claim above is sourced to Ai2's blog post rather than to a verified paper, and it should be read with exactly that level of confidence — plausible and worth watching, not yet pinned down the way the original Molmo and PixMo paper is.

The teaching point to carry out of this lecture is the same one Volume V returns to repeatedly: pointing and grounding deserve their own line in any evaluation, separate from ordinary description accuracy, precisely because a model can be fluent and ungrounded at the same time, and a report that only shows you captioning scores has quietly declined to answer the question this lecture asks.