# Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities

arXiv: https://arxiv.org/abs/2505.02567 · HTML: https://mlsanity.com/paper/2505.02567/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- Builds on 293 of its 397 references that are in the map.
- First posted on arXiv 5 May 2025.

## Builds on (most cited first, 30 of 293)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)

## Built on by (most cited first, 30 of 85)

- [Ovis-U1 Technical Report](https://mlsanity.com/paper/2506.23044/) (2025)
- [Reconstruction Alignment Improves Unified Multimodal Models](https://mlsanity.com/paper/2509.07295/) (2025)
- [AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration](https://mlsanity.com/paper/2510.10395/) (2025)
- [Mind-Brush: Integrating Agentic Cognitive Search and Reasoning into Image Generation](https://mlsanity.com/paper/2602.01756/) (2026)
- [Mull-Tokens: Modality-Agnostic Latent Thinking](https://mlsanity.com/paper/2512.10941/) (2025)
- [Ultralytics YOLO Evolution: An Overview of YOLO27, YOLO26, YOLO11, YOLOv8, and YOLOv5 Object Detectors for Computer Vision and Pattern Recognition](https://mlsanity.com/paper/2510.09653/) (2025)
- [Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images](https://mlsanity.com/paper/2511.07222/) (2025)
- [ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generation](https://mlsanity.com/paper/2511.11483/) (2025)
- [Object Detection with Multimodal Large Vision-Language Models: An In-depth Review](https://mlsanity.com/paper/2508.19294/) (2025)
- [Unified Multimodal Models as Auto-Encoders](https://mlsanity.com/paper/2509.09666/) (2025)
- [UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation](https://mlsanity.com/paper/2603.23500/) (2026)
- [MM-ACT: Learn from Multimodal Parallel Generation to Act](https://mlsanity.com/paper/2512.00975/) (2025)
- [Lance: Unified Multimodal Modeling by Multi-Task Synergy](https://mlsanity.com/paper/2605.18678/) (2026)
- [UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2506.17202/) (2025)
- [HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2511.20520/) (2025)
- [Language-Conditioned World Modeling for Visual Navigation](https://mlsanity.com/paper/2603.26741/) (2026)
- [Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model](https://mlsanity.com/paper/2607.24904/) (2026)
- [MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generation](https://mlsanity.com/paper/2511.18262/) (2025)
- [COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligence](https://mlsanity.com/paper/2512.04563/) (2025)
- [UEval: A Benchmark for Unified Multimodal Generation](https://mlsanity.com/paper/2601.22155/) (2026)
- [Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions](https://mlsanity.com/paper/2602.13013/) (2026)
- [DuetSVG: Unified Multimodal SVG Generation with Internal Visual Guidance](https://mlsanity.com/paper/2512.10894/) (2025)
- [Unified Personalized Understanding, Generating and Editing](https://mlsanity.com/paper/2601.06965/) (2026)
- [Toward Native Multimodal Modeling: A Roadmap](https://mlsanity.com/paper/2605.25343/) (2026)
- [GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Models](https://mlsanity.com/paper/2511.11134/) (2025)
- [Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Games](https://mlsanity.com/paper/2604.11741/) (2026)
- [ActiveMimic: Egocentric Video Pretraining with Active Perception](https://mlsanity.com/paper/2606.06194/) (2026)
- [EditCast3D: Single-Frame-Guided 3D Editing with Video Propagation and View Selection](https://mlsanity.com/paper/2510.13652/) (2025)
- [ConsistCompose: Unified Multimodal Layout Control for Image Composition](https://mlsanity.com/paper/2511.18333/) (2025)
- [Dynin-Omni: Omnimodal Unified Large Diffusion Language Model](https://mlsanity.com/paper/2604.00007/) (2026)

## Abstract (arXiv)

Recent years have seen remarkable progress in both multimodal understanding models and image generation models. Despite their respective successes, these two domains have evolved independently, leading to distinct architectural paradigms: While autoregressive-based architectures have dominated multimodal understanding, diffusion-based models have become the cornerstone of image generation. Recently, there has been growing interest in developing unified frameworks that integrate these tasks. The emergence of GPT-4o's new capabilities exemplifies this trend, highlighting the potential for unification. However, the architectural differences between the two domains pose significant challenges. To provide a clear overview of current efforts toward unification, we present a comprehensive survey aimed at guiding future research. First, we introduce the foundational concepts and recent advancements in multimodal understanding and text-to-image generation models. Next, we review existing unified models, categorizing them into three main architectural paradigms: diffusion-based, autoregressive-based, and hybrid approaches that fuse autoregressive and diffusion mechanisms. For each category, we analyze the structural designs and innovations introduced by related works. Additionally, we compile datasets and benchmarks tailored for unified models, offering resources for future exploration. Finally, we discuss the key challenges facing this nascent field, including tokenization strategy, cross-modal attention, and data. As this area is still in its early stages, we anticipate rapid advancements and will regularly update this survey. Our goal is to inspire further research and provide a valuable reference for the community. The references associated with this survey are available on GitHub (https://github.com/AIDC-AI/Awesome-Unified-Multimodal-Models).