Volume IV · Understanding and Generating

Seeing in Time

Every mechanism in the last eight lectures answered the same question: given one picture, how does a model turn it into tokens. A video is not one picture. It is many pictures, in an order that matters, over a duration that matters — and nothing about the machinery built so far tells you what to do with either fact.

The obvious move, and its cost

The dominant older pattern is the one you would invent first if nobody had told you otherwise. Sample a handful of frames from the video at some fixed rate, hand each one to an ordinary image encoder — the same CLIP- or ViT-style machinery from Lecture 1 and Lecture 2, run frame by frame, with no awareness that neighbouring frames are related at all — and pool the resulting features, spatially and temporally, before the language model ever sees them. This per-frame-encode-then-pool pattern was the approach taken by the earliest video-LLM systems, and it has the enormous appeal of requiring no new architecture: it reuses an image encoder outright and adds only a pooling step on top.

The cost is not subtle, and it is worth stating precisely rather than gesturing at it. Pooling averages features together, and averaging destroys whatever was distinct between the frames being averaged. If a hand closes around a cup between frame 4 and frame 5, a temporal pool that blends those two frames' features is, by construction, throwing away the very information that made them two frames instead of one. Fine-grained temporal detail is not incidentally lost by this design — it is the direct, mechanical consequence of the pooling operation itself.

Making time explicit

The newer pattern does not try to fix pooling; it tries to stop relying on it. Rather than leaving temporal information to be inferred — or averaged away — from an unordered bag of frame features, these approaches inject an explicit temporal signal directly into the model's inputs. A recent survey of video temporal grounding methods describes this shift in some detail, and it is worth naming two concrete instances it discusses, while being honest about how firmly this lecture can stand behind the specifics: that survey was matched to this lecture by title and identifier during research rather than fetched and read in full, so treat the finer mechanism claims that follow as a reasonable characterisation rather than a verbatim account.

The first instance adds dedicated temporal tokens directly to the model's vocabulary — special tokens whose entire job is to mark position in time, sitting alongside ordinary text and image tokens rather than being inferred from them. The second is stranger and, in a way, more direct: numeric frame indices are overlaid onto the video frames themselves, so that the language model reads sequence order the same way it reads any other text baked into an image — by looking at it, not by being told about it through a side channel.

Time built into position itself

The most recent pattern goes a step further than adding a temporal signal alongside the existing machinery — it rebuilds the position-encoding scheme to carry time natively, which is the approach this track has already covered in the Qwen2.5-VL and Qwen3-VL survey lecture. Rather than treating "when in the video" as an LLM-side add-on bolted on after encoding, Qwen3-VL's Interleaved-MRoPE models spatial and temporal structure jointly, inside the position-encoding scheme itself, with absolute and relative time encoded directly rather than left for a downstream layer to reconstruct. Its explicit text-based timestamp alignment is aimed specifically at video temporal grounding at the precision of individual seconds — the kind of question a pooled representation has already made impossible to answer by the time it reaches the language model.

Read across the three patterns in order and a trend is visible: each one moves the temporal signal earlier and makes it more structural. Pooling handles time as an afterthought, applied once encoding is already done. Explicit temporal tokens and burned-in timestamps handle time as an addition, sitting alongside the ordinary tokens. Position encoding native to time handles it as a first-class part of how the model represents any token at all, video or not.

Why this dimension resists a clean answer

Part of the reason no single pattern has won is that "temporal detail" is not one thing to optimise for. A question like "did the cup get picked up" needs to notice one transition between two frames. A question like "how long did the meeting run" needs duration, which is a property of the whole clip rather than any single frame boundary. A question like "what happened right after the door opened" needs both order and a notion of "right after" that survives however the frames were sampled. Pooling degrades gracefully on none of these. Explicit tokens and timestamps help with the first and third at some added cost. Native temporal position encoding is the only one of the three built to handle duration and fine-grained order at once — which is presumably why it is where the frontier line covered in Lecture 7 has moved, even though the survey literature has not yet declared the older approaches obsolete.

The honest takeaway to carry forward is not a rule but a question. When a report says its model "understands video," the useful follow-up is not whether that is true in the abstract — it almost certainly is, to some degree — but which of these three tokenization strategies it used to get there, and what kind of temporal question that strategy was actually built to answer.