mlsanityData to 7 Oct 2026

Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models

arXiv 2603.06043 ↗First posted 6 Mar 2026cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Recently, unified multimodal models (UMMs) have made remarkable progress in integrating visual understanding and generation, demonstrating strong potential for complex text-to-image (T2I) tasks. Despite their theoretical promise, a persistent capability gap exists: UMMs typically exhibit superior visual understanding but comparatively weaker generative capabilities. This discrepancy arises largely from the intrinsic decoupling between the understanding and generation processes. While a UMM can accurately interpret fine-grained visual details, it often struggles to produce semantically coherent images from complex textual prompts. To address this challenge, we explore UMMs' internal understanding capability to enhance generation quality. We propose a token-level intrinsic text-image alignment reward mechanism, GvU, enabling the UMM to act simultaneously as teacher and student: it evaluates its own outputs using the understanding branch to guide the generations accordingly. Building upon this, we design a self-supervised reinforcement learning framework, allowing UMMs to iteratively improve their generation quality through understanding-based intrinsic reward signals--without reliance on external supervision. Experimental results show that our method substantially boosts UMMs' generation, which in turn strengthens their fine-grained visual understanding, narrowing the capability gap between UMMs' visual understanding and generation.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 50 of its 60 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

50 of 60 references.

References · 50

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
38 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  5. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  6. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  7. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  8. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  9. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  10. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  11. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  12. 2024Self-Rewarding Language Modelscited by 654
  13. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  14. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  15. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  16. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  17. 2024OmniGen: Unified Image Generationcited by 388
  18. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
  19. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  20. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  21. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  22. 2023Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspectivecited by 276
  23. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  24. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  25. 2025Transfer between Modalities with MetaQueriescited by 231
  26. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  27. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  28. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  29. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  30. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  31. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  32. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
  33. 2025Unified Autoregressive Visual Generation and Understanding with Continuous Tokenscited by 51
  34. 2025GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learningcited by 51
  35. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 48
  36. 2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learningcited by 23
  37. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Modelscited by 19
  38. 2025Turning Internal Gap into Self-Improvement: Promoting the Generation-Understanding Unification in MLLMscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.