Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System
Not on the map. Why?
Abstract (arXiv)
While unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model, functional unification does not guarantee learning synergy: the two objectives may reinforce each other, compete for capacity, or merely coexist. We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting without pretrained vision priors. At the representation level, we find that each objective provides useful signal to the other: generation enriches the visual features learned for understanding, while understanding strengthens vision--language alignment for generation. However, when both objectives are forced through the same computation path, one tends to dominate. A task-decoupled architecture that specializes conflicting visual computation while preserving semantic interaction avoids this asymmetric degradation. At the task level, through three case studies, we find positive bidirectional transfer when understanding and generation tasks rely on shared knowledge. At the system level, we show that an end-to-end UMM outperforms a matched planner--executor pipeline on complex tasks that explicitly require both image understanding and generation. Together, these results show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 77 of its 87 references that are in the map. First posted on arXiv 1 Sep 2026.
Builds on
77 of 87 references.
References · 77
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2025Qwen3 Technical Reportcited by 8,195
- 2013Auto-Encoding Variational Bayescited by 3,980
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
38 more
- 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
- 2024Depth Anything V2cited by 1,385
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
- 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
- 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
- 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
- 2016A Diagram Is Worth A Dozen Imagescited by 793
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
- 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
- 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
- 2017Large Batch Training of Convolutional Networkscited by 529
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
- 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
- 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
- 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
- 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
- 2021InfographicVQAcited by 430
- 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
- 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
- 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
- 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
- 2025Transfer between Modalities with MetaQueriescited by 231
- 2025HPSv3: Towards Wide-Spectrum Human Preference Scorecited by 196
- 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
- 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 159
- 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
- 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
- 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
- 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
- 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
- 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 110
- 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 97
50 most cited of 77 All 77 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.