mlsanityData to 7 Oct 2026

Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

arXiv 2408.11039 ↗First posted 20 Aug 2024cs.AI cs.CVcited by 468

Not on the map. Why?

Cited by 94 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Transfusion, a recipe for training a multi-modal model over discrete and continuous data. Transfusion combines the language modeling loss function (next token prediction) with diffusion to train a single transformer over mixed-modality sequences. We pretrain multiple Transfusion models up to 7B parameters from scratch on a mixture of text and image data, establishing scaling laws with respect to a variety of uni- and cross-modal benchmarks. Our experiments show that Transfusion scales significantly better than quantizing images and training a language model over discrete image tokens. By introducing modality-specific encoding and decoding layers, we can further improve the performance of Transfusion models, and even compress each image to just 16 patches. We further demonstrate that scaling our Transfusion recipe to 7B parameters and 2T multi-modal tokens produces a model that can generate images and text on a par with similar scale diffusion models and language models, reaping the benefits of both worlds.

Key facts. Cited by 468 papers in the map (data through 7 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

41 of 50 references.

References · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
29 more
  1. 2022Flow Matching for Generative Modelingcited by 5,426
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  13. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  14. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  15. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  16. 2020GLU Variants Improve Transformercited by 1,724
  17. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  18. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  19. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  20. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  21. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  22. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  23. 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 321
  24. 2024Discrete Flow Matchingcited by 301
  25. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  26. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  27. 2023Generating Images with Multimodal Language Modelscited by 259
  28. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  29. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 69; 2025: 234; 2026: 165.

468 papers.

Sort:

By area:

  1. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  2. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  3. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  4. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  5. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  6. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  7. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  8. 2024OmniGen: Unified Image Generationcited by 388
  9. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
  10. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  11. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  12. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
38 more
  1. 2025Transfer between Modalities with MetaQueriescited by 231
  2. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  3. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 204
  4. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  5. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  6. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  7. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  8. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  9. 2024In-Context LoRA for Diffusion Transformerscited by 155
  10. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  11. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  12. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  13. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  14. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 142
  15. 2025HunyuanImage 3.0 Technical Reportcited by 138
  16. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  17. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  18. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 124
  19. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  20. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  21. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  22. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  23. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 106
  24. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  25. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  26. 2024UniReal: Universal Image Generation and Editing via Learning Real-world Dynamicscited by 100
  27. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 97
  28. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
  29. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 94
  30. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 93
  31. 2025PixelFlow: Pixel-Space Generative Models with Flowcited by 91
  32. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 91
  33. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  34. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  35. 2024LLaMA-Mesh: Unifying 3D Mesh Generation with Language Modelscited by 87
  36. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 85
  37. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actionscited by 85
  38. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 85

50 most cited of 468 All 468 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models
  2. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders
  3. 2026Abra: Scaling Diffusion Image Training
  4. 2022TransFusion: Transcribing Speech with Multinomial Diffusion
  5. 2023MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation
  6. 2025Discrete Diffusion Models with MLLMs for Unified Medical Multimodal Generation
  7. 2025Med-Art: Diffusion Transformer for 2D Medical Text-to-Image Generation
  8. 2025LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding