mlsanityData to 14 Oct 2026

Unified Multimodal Models as Auto-Encoders

arXiv 2509.09666 ↗First posted 11 Sep 2025cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Image-to-text (I2T) understanding and text-to-image (T2I) generation are two fundamental, important yet traditionally isolated multimodal tasks. Despite their intrinsic connection, existing approaches typically optimize them independently, missing the opportunity for mutual enhancement. In this paper, we argue that the both tasks can be connected under a shared Auto-Encoder perspective, where text serves as the intermediate latent representation bridging the two directions - encoding images into textual semantics (I2T) and decoding text back into images (T2I). Our key insight is that if the encoder truly "understands" the image, it should capture all essential structure, and if the decoder truly "understands" the text, it should recover that structure faithfully. Building upon this principle, we propose Unified-GRPO, a post-training method based on reinforcement learning that jointly optimizes both modules through reconstructive rewards, maximizing the semantic consistency between the input and the generated images. Under this reconstruction objective, the encoder is encouraged to extract as much accurate and comprehensive semantic information from the input image to maximize reconstruction quality, while the decoder is simultaneously optimized to generate conditioned on the encoder's prior, enabling a self-evolving improvement. Empirically, we find that using text as the intermediate representation and training under a reconstructive RL paradigm effectively benefits both I2T and T2I. The I2T module gains stronger fine-grained visual perception, such as small-object recognition, grounding, etc, while its dense embeddings and language priors, in turn, provide richer semantic signals that improve T2I fidelity and complex instruction following. These results demonstrate that the reconstructive RL establishes a mutually reinforcing cross-modal synergy within the auto-encoding framework.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 61 of its 70 references that are in the map. First posted on arXiv 11 Sep 2025.

Builds on

61 of 70 references.

References · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
38 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  3. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  4. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  5. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  6. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  7. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  8. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  9. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  10. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  11. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  12. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  13. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  14. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  15. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  16. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  17. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 413
  18. 2024OmniGen: Unified Image Generationcited by 394
  19. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 376
  20. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  21. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
  22. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 260
  23. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 256
  24. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 253
  25. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  26. 2025Transfer between Modalities with MetaQueriescited by 236
  27. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 204
  28. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  29. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  30. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 171
  31. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 166
  32. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  33. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 156
  34. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  35. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  36. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  37. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 100
  38. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 95

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

By area:

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization
  2. 2026Unified Text-Image Generation with Weakness-Targeted Post-Training
  3. 2026Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models
  4. 2026ReasonUni: Reasoning-Augmented Reconstruction Advances Unified Multimodal Models
  5. 2025ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
  6. 2026FiRe: Fine-grained Multimodal Reasoning for Enhanced Image Generation
  7. 2024VersaT2I: Improving Text-to-Image Models with Versatile Reward
  8. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models