Transferability Between Understanding and Generation in Unified Multimodal Models
Not on the map. Why?
Abstract (arXiv)
Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate boldsymbolmathsftransferability in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 94 of its 109 references that are in the map. First posted on arXiv 5 Jul 2026.
Builds on
94 of 109 references.
References · 94
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2023GPT-4 Technical Reportcited by 18,132
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2020Denoising Diffusion Implicit Modelscited by 9,655
- 2023Visual Instruction Tuningcited by 7,792
- 2022Flow Matching for Generative Modelingcited by 5,426
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2017Neural Discrete Representation Learningcited by 4,650
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
38 more
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
- 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
- 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
- 2016Improved Techniques for Training GANscited by 2,938
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
- 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
- 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
- 2023Training Diffusion Models with Reinforcement Learningcited by 939
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
- 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
- 2024Simple and Effective Masked Diffusion Language Modelscited by 800
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
- 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
- 2024Emu3: Next-Token Prediction is All You Needcited by 761
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
- 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
- 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
- 2025Qwen3-Omni Technical Reportcited by 451
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
- 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
- 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
- 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
- 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
- 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
- 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
- 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
- 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
50 most cited of 94 All 94 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.