mlsanityData to 7 Oct 2026

Emu3: Next-Token Prediction is All You Need

arXiv 2409.18869 ↗First posted 27 Sep 2024cs.CVcited by 761

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

While next-token prediction is considered a promising path towards artificial general intelligence, it has struggled to excel in multimodal tasks, which are still dominated by diffusion models (e.g., Stable Diffusion) and compositional approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction. By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences. Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship models such as SDXL and LLaVA-1.6, while eliminating the need for diffusion or compositional architectures. Emu3 is also capable of generating high-fidelity video via predicting the next token in a video sequence. We simplify complex multimodal model designs by converging on a singular focus: tokens, unlocking great potential for scaling both during training and inference. Our results demonstrate that next-token prediction is a promising path towards building general multimodal intelligence beyond language. We open-source key techniques and models to support further research in this direction.

Key facts. Cited by 761 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 86 of its 102 references that are in the map. First posted on arXiv 27 Sep 2024.

Builds on

86 of 102 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding14 May
  2. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment8 Mar
  3. 2023Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following28 Nov
  4. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment17 Oct
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 81

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Scalable Diffusion Models with Transformerscited by 5,931
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  13. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  14. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  15. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  16. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  17. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  18. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  19. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  20. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  21. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  22. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  23. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  24. 2020GLU Variants Improve Transformercited by 1,724
  25. 2019Towards VQA Models That Can Readcited by 1,696
  26. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  27. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  28. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  29. 2019Root Mean Square Layer Normalizationcited by 1,425
  30. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  31. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  32. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  33. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  34. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  35. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,199
  36. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  37. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  38. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 71; 2025: 390; 2026: 300.

761 papers.

Sort:

By area:

  1. 2025Qwen-Image Technical Reportcited by 973
  2. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  3. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  4. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  5. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  6. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  7. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
  8. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  9. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
  10. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  11. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  12. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
38 more
  1. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  2. 2025Transfer between Modalities with MetaQueriescited by 231
  3. 2025Motus: A Unified Latent Action World Modelcited by 231
  4. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  5. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  6. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  7. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  8. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  9. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  10. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  11. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 157
  12. 2024In-Context LoRA for Diffusion Transformerscited by 155
  13. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  14. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  15. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  16. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  17. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  18. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  19. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  20. 2025Unified Vision-Language-Action Modelcited by 128
  21. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  22. 2024Randomized Autoregressive Visual Generationcited by 125
  23. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  24. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  25. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  26. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  27. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  28. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 102
  29. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  30. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 97
  31. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  32. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
  33. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  34. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 94
  35. 2025Lumina-Image 2.0: A Unified and Efficient Image Generative Frameworkcited by 93
  36. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 93
  37. 2025LongCat-Image Technical Reportcited by 92
  38. 2025PixelFlow: Pixel-Space Generative Models with Flowcited by 91

50 most cited of 761 All 761 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Emu: Generative Pretraining in Multimodality
  2. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
  3. 2024MIO: A Foundation Model on Multimodal Tokens
  4. 2025Token Sequence Compression for Efficient Multimodal Computing
  5. 2026Efficient Token Pruning for LLaDA-V
  6. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
  7. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMs
  8. 2026TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs