mlsanityData to 14 Oct 2026

Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities

arXiv 2505.02567 ↗First posted 5 May 2025cs.CVcited by 85

Not on the map. Why?

Abstract (arXiv)

Recent years have seen remarkable progress in both multimodal understanding models and image generation models. Despite their respective successes, these two domains have evolved independently, leading to distinct architectural paradigms: While autoregressive-based architectures have dominated multimodal understanding, diffusion-based models have become the cornerstone of image generation. Recently, there has been growing interest in developing unified frameworks that integrate these tasks. The emergence of GPT-4o's new capabilities exemplifies this trend, highlighting the potential for unification. However, the architectural differences between the two domains pose significant challenges. To provide a clear overview of current efforts toward unification, we present a comprehensive survey aimed at guiding future research. First, we introduce the foundational concepts and recent advancements in multimodal understanding and text-to-image generation models. Next, we review existing unified models, categorizing them into three main architectural paradigms: diffusion-based, autoregressive-based, and hybrid approaches that fuse autoregressive and diffusion mechanisms. For each category, we analyze the structural designs and innovations introduced by related works. Additionally, we compile datasets and benchmarks tailored for unified models, offering resources for future exploration. Finally, we discuss the key challenges facing this nascent field, including tokenization strategy, cross-modal attention, and data. As this area is still in its early stages, we anticipate rapid advancements and will regularly update this survey. Our goal is to inspire further research and provide a valuable reference for the community. The references associated with this survey are available on GitHub (https://github.com/AIDC-AI/Awesome-Unified-Multimodal-Models).

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Builds on 293 of its 397 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

293 of 397 references.

References · 293

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2020Denoising Diffusion Implicit Modelscited by 9,781
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  12. 2023Segment Anythingcited by 8,234
38 more
  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2022Generative Adversarial Networkscited by 5,627
  8. 2025Qwen2.5-VL Technical Reportcited by 5,587
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  11. 2017Neural Discrete Representation Learningcited by 4,712
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  13. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  14. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  15. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  16. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  17. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  18. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  19. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  20. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  21. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  22. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  23. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  24. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  25. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  26. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  27. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  28. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  29. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  30. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  31. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  32. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  33. 2023A Survey of Large Language Modelscited by 2,576
  34. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  35. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  36. 2015VQA: Visual Question Answeringcited by 2,506
  37. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  38. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222

50 most cited of 293 All 293 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 58.

85 papers.

Sort:

By area:

  1. 2025Ovis-U1 Technical Reportcited by 67
  2. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 49
  3. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 27
  4. 2026Mind-Brush: Integrating Agentic Cognitive Search and Reasoning into Image Generationcited by 23
  5. 2025Mull-Tokens: Modality-Agnostic Latent Thinkingcited by 19
  6. 2025Ultralytics YOLO Evolution: An Overview of YOLO27, YOLO26, YOLO11, YOLOv8, and YOLOv5 Object Detectors for Computer Vision and Pattern Recognitioncited by 18
  7. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 17
  8. 2025ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generationcited by 17
  9. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  10. 2025Unified Multimodal Models as Auto-Encoderscited by 13
  11. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generationcited by 13
  12. 2025MM-ACT: Learn from Multimodal Parallel Generation to Actcited by 12
38 more
  1. 2026Lance: Unified Multimodal Modeling by Multi-Task Synergycited by 12
  2. 2025UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generationcited by 11
  3. 2025HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generationcited by 11
  4. 2026Language-Conditioned World Modeling for Visual Navigationcited by 10
  5. 2026Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Modelcited by 10
  6. 2025MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generationcited by 9
  7. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  8. 2026UEval: A Benchmark for Unified Multimodal Generationcited by 9
  9. 2026Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructionscited by 9
  10. 2025DuetSVG: Unified Multimodal SVG Generation with Internal Visual Guidancecited by 8
  11. 2026Unified Personalized Understanding, Generating and Editingcited by 8
  12. 2026Toward Native Multimodal Modeling: A Roadmapcited by 7
  13. 2025GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Modelscited by 6
  14. 2026Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Gamescited by 6
  15. 2026ActiveMimic: Egocentric Video Pretraining with Active Perceptioncited by 6
  16. 2025EditCast3D: Single-Frame-Guided 3D Editing with Video Propagation and View Selectioncited by 5
  17. 2025ConsistCompose: Unified Multimodal Layout Control for Image Compositioncited by 5
  18. 2026Dynin-Omni: Omnimodal Unified Large Diffusion Language Modelcited by 5
  19. 2026Can Vision-Language Models Think from the Sky? Unifying UAV Reasoning and Generationcited by 5
  20. 2026Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Modelscited by 5
  21. 2025Speculative Coupled Decoding for Training-Free Lossless Acceleration of Autoregressive Visual Generationcited by 4
  22. 2025Understanding and Harnessing Sparsity in Unified Multimodal Modelscited by 4
  23. 2026PresentAgent-2: Towards Generalist Multimodal Presentation Agentscited by 4
  24. 2026Scaling GUI Agents with Visual State Transitionscited by 4
  25. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 3
  26. 2026CyCLeGen: Cycle-Consistent Layout Prediction and Image Generation in Vision Foundation Modelscited by 3
  27. 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Modelscited by 3
  28. 2026UniGenDet: A Unified Generative-Discriminative Framework for Co-Evolutionary Image Generation and Generated Image Detectioncited by 3
  29. 2026ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Bothcited by 3
  30. 2026Transferability Between Understanding and Generation in Unified Multimodal Modelscited by 3
  31. 2026Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editingcited by 3
  32. 2025Caption Injection for Optimization in Generative Search Enginecited by 2
  33. 2025What Happens Next? Next Scene Prediction with a Unified Video Modelcited by 2
  34. 2026IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translationcited by 2
  35. 2026TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-trainingcited by 2
  36. 2026PaintBench: Deterministic Evaluation of Precise Visual Editingcited by 2
  37. 2026Event-VLA: Action-Conditioned Event Fusion for Robust Vision-Language-Action Modelcited by 2
  38. 2025M-PACE: Mother Child Framework for Multimodal Compliancecited by 1

50 most cited of 85 All 85 (JSON)