mlsanityData to 7 Oct 2026

MonoFormer: One Transformer for Both Diffusion and Autoregression

arXiv 2409.16280 ↗First posted 24 Sep 2024cs.CVcited by 45

Not on the map. Why?

Abstract (arXiv)

Most existing multimodality methods use separate backbones for autoregression-based discrete text generation and diffusion-based continuous visual generation, or the same backbone by discretizing the visual data to use autoregression for both text and visual generation. In this paper, we propose to study a simple idea: share one transformer for both autoregression and diffusion. The feasibility comes from two main aspects: (i) Transformer is successfully applied to diffusion for visual generation, and (ii) transformer training for autoregression and diffusion is very similar, and the difference merely lies in that diffusion uses bidirectional attention mask and autoregression uses causal attention mask. Experimental results show that our approach achieves comparable image generation performance to current state-of-the-art methods as well as maintains the text generation capability. The project is publicly available at https://monoformer.github.io/.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 45 of its 53 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

45 of 53 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
33 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  13. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  14. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  15. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  16. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  17. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  18. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  19. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  20. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  21. 2018Image Transformercited by 770
  22. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  23. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  24. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  25. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  26. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  27. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  28. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  29. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  30. 2023Generating Images with Multimodal Language Modelscited by 259
  31. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  32. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  33. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 26; 2026: 9.

45 papers.

Sort:

By area:

  1. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  2. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  3. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  4. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  5. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  6. 2025Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generationcited by 71
  7. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  8. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 52
  9. 2024ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformercited by 45
  10. 2024Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Headscited by 44
  11. 2024DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generationcited by 32
  12. 2024Causal Diffusion Transformers for Generative Modelingcited by 31
33 more
  1. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 23
  2. 2024MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modelingcited by 19
  3. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Modelscited by 19
  4. 2025PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Modelscited by 18
  5. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  6. 2025OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flowscited by 17
  7. 2025UGen: Unified Autoregressive Multimodal Model with Progressive Vocabulary Learningcited by 16
  8. 2025ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedbackcited by 15
  9. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  10. 2026LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokenscited by 10
  11. 2024Divot: Diffusion Powers Video Tokenizer for Comprehension and Generationcited by 8
  12. 2025Planned Diffusioncited by 8
  13. 2024MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistantscited by 6
  14. 2025Open Multimodal Retrieval-Augmented Factual Image Generationcited by 6
  15. 2026Endogenous Reprompting: Self-Evolving Cognitive Alignment for Unified Multimodal Modelscited by 6
  16. 2026CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generationcited by 6
  17. 2025Direction-Aware Diagonal Autoregressive Image Generationcited by 5
  18. 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modelingcited by 5
  19. 2025End-to-End Multi-Modal Diffusion Mambacited by 3
  20. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  21. 2025TV2TV: A Unified Framework for Interleaved Language and Video Generationcited by 3
  22. 2026CoLoGen: Progressive Learning of Concept-Localization Duality for Unified Image Generationcited by 3
  23. 2026CyCLeGen: Cycle-Consistent Layout Prediction and Image Generation in Vision Foundation Modelscited by 3
  24. 2026Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Trainingcited by 3
  25. 2025Envision: Benchmarking Unified Understanding & Generation for Causal World Process Insightscited by 2
  26. 2025UniCMs: A Unified Consistency Model For Efficient Multimodal Generation and Understandingcited by 1
  27. 2025UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Modelscited by 1
  28. 2025Generative AI Meets 6G and Beyond: Diffusion Models for Semantic Communicationscited by 1
  29. 2025ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interactioncited by 1
  30. 2025From Perception to Punchline: Empowering VLM with the Art of In-the-wild Memecited by 1
  31. 2025D2C: Unlocking the Potential of Continuous Autoregressive Image Generation with Discrete Tokenscited by 0
  32. 2025Nested AutoRegressive Modelscited by 0
  33. 2026DIVA: Harnessing the Representation Divergence in Unified Multimodal Models for Mutual Reinforcementcited by 0