Volume IV · Understanding and Generating

One Model, Both Directions

Ask a model to describe a photograph and ask it to paint one, and you are asking for two different skills wearing the same word. Understanding wants to compress a picture down to what matters; generation wants to keep every detail that might matter later. Three 2024 and 2025 models each resolve that tension a different way, while still sharing one transformer underneath.

Show-o: hybrid, autoregressive plus diffusion

Show-o, from the Show Lab at NUS, published in August 2024, takes the position that the two jobs need two different generative mechanisms living inside one model. Rather than being purely autoregressive throughout, Show-o unifies autoregressive modelling — the mechanism this entire track has assumed for text — with discrete diffusion modelling, used specifically for image tokens, inside a single transformer that adaptively switches mode depending on which modality it is currently producing. Text is generated left to right, one token at a time, exactly as in every prior lecture; images are generated by the iterative denoising process diffusion models use, over a fixed set of discrete image tokens.

One set of shared weights, working this way, supports visual question answering, text-to-image generation, text-guided inpainting and extrapolation, and mixed-modality generation — and reportedly matches or beats separate specialist models of equal or larger size at each of those tasks individually. Frame this model precisely as this lecture's hybrid position: autoregressive plus diffusion, in one model, switching by modality.

Janus and Janus-Pro: one backbone, two front doors

DeepSeek's Janus, published in October 2024, and its refinement Janus-Pro, published in January 2025, take a different position: keep one shared autoregressive transformer as the single point of unification, but stop asking one visual encoder to serve both understanding and generation. Separate, independent encoding paths feed the same unified transformer.

For understanding, a SigLIP encoder — the same family of vision encoder this track met in Lecture 1 — extracts high-dimensional semantic features, flattens them into a sequence, and maps them through an understanding adaptor into the transformer's input space. For generation, images are instead converted into discrete VQ tokens, in the same VQGAN lineage Lecture 2 and Lecture 6 both drew on, through a wholly separate tokenizer and adaptor pathway. Both streams converge into one shared transformer that predicts the next token regardless of which task is currently underway.

The paper's core finding is precise and worth stating exactly: a single shared visual encoder creates real tension because understanding wants high-level semantic features — what is in the picture — while generation wants low-level, detail-preserving features — what exactly does each pixel look like. Decoupling the encoding paths resolves that tension without requiring two entirely separate models. Frame this as this lecture's middle position: one shared backbone, two separate front doors.

Emu3: nothing but next-token prediction

Emu3, from the Beijing Academy of AI, published in September 2024, takes the purest position on this spectrum. Images, text, and video are all tokenized into one single discrete vocabulary, and a single transformer is trained from scratch purely with next-token prediction over mixed multimodal sequences — no diffusion component anywhere in the architecture, and no compositional multi-module design of the kind Janus-Pro just built. Image generation happens simply by predicting the next vision token in sequence; video generation happens by predicting the next token in a video-token sequence, extending the same single mechanism into a third modality neither Show-o nor Janus-Pro addresses directly.

Emu3 reports outperforming SDXL, LLaVA-1.6, and OpenSora-1.2 despite this architectural simplicity — specialist diffusion and specialist video models, beaten by one model whose entire mechanism is the same next-token objective this whole track opened with in Lecture 1. Frame this as the lecture's far end: just tokens and next-token prediction, nothing else, including for video.

The spectrum, stated plainly

Close this lecture the way it opened: as a three-point spectrum, not three unrelated models. Show-o is a hybrid of two generative paradigms living in one model. Janus-Pro shares one transformer but still specialises its encoders by task. Emu3 is the single, unadorned "everything is a token, predict the next one" position, extended even to video — the purest answer this lecture has to the tension it opened with.