mlsanityData to 7 Oct 2026

Transferability Between Understanding and Generation in Unified Multimodal Models

arXiv 2607.04423 ↗First posted 5 Jul 2026cs.CV cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate boldsymbolmathsftransferability in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 94 of its 109 references that are in the map. First posted on arXiv 5 Jul 2026.

Builds on

94 of 109 references.

References · 94

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2022Flow Matching for Generative Modelingcited by 5,426
  10. 2025Qwen2.5-VL Technical Reportcited by 5,418
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
38 more
  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2016Improved Techniques for Training GANscited by 2,938
  7. 2025Qwen3-VL Technical Reportcited by 2,534
  8. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  13. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  14. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  15. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  16. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  17. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  18. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  19. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  20. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  21. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  22. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  23. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  24. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  25. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  26. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  27. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  28. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  29. 2025Qwen3-Omni Technical Reportcited by 451
  30. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  31. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  32. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  33. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  34. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
  35. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  36. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  37. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  38. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.