mlsanityData to 7 Oct 2026

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

arXiv 2609.01607 ↗First posted 1 Sep 2026cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

While unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model, functional unification does not guarantee learning synergy: the two objectives may reinforce each other, compete for capacity, or merely coexist. We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting without pretrained vision priors. At the representation level, we find that each objective provides useful signal to the other: generation enriches the visual features learned for understanding, while understanding strengthens vision--language alignment for generation. However, when both objectives are forced through the same computation path, one tends to dominate. A task-decoupled architecture that specializes conflicting visual computation while preserving semantic interaction avoids this asymmetric degradation. At the task level, through three case studies, we find positive bidirectional transfer when understanding and generation tasks rely on shared knowledge. At the system level, we show that an end-to-end UMM outperforms a matched planner--executor pipeline on complex tasks that explicitly require both image understanding and generation. Together, these results show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 77 of its 87 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

77 of 87 references.

References · 77

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2013Auto-Encoding Variational Bayescited by 3,980
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
38 more
  1. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  3. 2024Depth Anything V2cited by 1,385
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  6. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  7. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  8. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  9. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  10. 2016A Diagram Is Worth A Dozen Imagescited by 793
  11. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  12. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  13. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  14. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  15. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  16. 2017Large Batch Training of Convolutional Networkscited by 529
  17. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  18. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  19. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  20. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  21. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  22. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  23. 2021InfographicVQAcited by 430
  24. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  25. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
  26. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  27. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  28. 2025Transfer between Modalities with MetaQueriescited by 231
  29. 2025HPSv3: Towards Wide-Spectrum Human Preference Scorecited by 196
  30. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  31. 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 159
  32. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  33. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  34. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  35. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  36. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
  37. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 110
  38. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 97

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.