Unified Multimodal Models as Auto-Encoders
Not on the map. Why?
Abstract (arXiv)
Image-to-text (I2T) understanding and text-to-image (T2I) generation are two fundamental, important yet traditionally isolated multimodal tasks. Despite their intrinsic connection, existing approaches typically optimize them independently, missing the opportunity for mutual enhancement. In this paper, we argue that the both tasks can be connected under a shared Auto-Encoder perspective, where text serves as the intermediate latent representation bridging the two directions - encoding images into textual semantics (I2T) and decoding text back into images (T2I). Our key insight is that if the encoder truly "understands" the image, it should capture all essential structure, and if the decoder truly "understands" the text, it should recover that structure faithfully. Building upon this principle, we propose Unified-GRPO, a post-training method based on reinforcement learning that jointly optimizes both modules through reconstructive rewards, maximizing the semantic consistency between the input and the generated images. Under this reconstruction objective, the encoder is encouraged to extract as much accurate and comprehensive semantic information from the input image to maximize reconstruction quality, while the decoder is simultaneously optimized to generate conditioned on the encoder's prior, enabling a self-evolving improvement. Empirically, we find that using text as the intermediate representation and training under a reconstructive RL paradigm effectively benefits both I2T and T2I. The I2T module gains stronger fine-grained visual perception, such as small-object recognition, grounding, etc, while its dense embeddings and language priors, in turn, provide richer semantic signals that improve T2I fidelity and complex instruction following. These results demonstrate that the reconstructive RL establishes a mutually reinforcing cross-modal synergy within the auto-encoding framework.
Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 61 of its 70 references that are in the map. First posted on arXiv 11 Sep 2025.
Builds on
61 of 70 references.
References · 61
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2015VQA: Visual Question Answeringcited by 2,506
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
38 more
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
- 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
- 2023Training Diffusion Models with Reinforcement Learningcited by 982
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
- 2024Emu3: Next-Token Prediction is All You Needcited by 781
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
- 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
- 2024Autoregressive Image Generation without Vector Quantizationcited by 651
- 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
- 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
- 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 413
- 2024OmniGen: Unified Image Generationcited by 394
- 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 376
- 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
- 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
- 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 260
- 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 256
- 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 253
- 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
- 2025Transfer between Modalities with MetaQueriescited by 236
- 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 204
- 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
- 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
- 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 171
- 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 166
- 2023Holistic Evaluation of Text-To-Image Modelscited by 160
- 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 156
- 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
- 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
- 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
- 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 100
- 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 95
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2025: 2; 2026: 11.
13 papers.
Sort:
By area:
- 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 56
- 2026Mind-Brush: Integrating Agentic Cognitive Search and Reasoning into Image Generationcited by 23
- 2025Seeing Before Reasoning: A Unified Framework for Generalizable and Explainable Fake Image Detectioncited by 20
- 2026Think-Then-Generate: Reasoning-Aware Text-to-Image Diffusion with LLM Encoderscited by 15
- 2026UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Modelscited by 7
- 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modelingcited by 5
- 2026Dreaming in Flow: Generative Grounding Feedback for Self-Evolving Unified Multimodal Modelscited by 3
- 2026Advances in GRPO for Generation Models: A Surveycited by 2
- 2026WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modelingcited by 0
- 2026STBridge: Shared-Target Alignment for Bridging Understanding and Generation in UMMscited by 0
- 2026UniWorld-Design: From Pixel Generation to Layer-Native Designcited by 0
- 2026HounsWorld: A Multimodal World Model for Hidden Patient-State Readout, Reconstruction, and Simulationcited by 0
1 more
- 2026Editable Visual Designcited by 0
- 2026Dreaming in Flow: Generative Grounding Feedback for Self-Evolving Unified Multimodal Models8 Sep 2026
- 2026Editable Visual Design3 Sep 2026
- 2026HounsWorld: A Multimodal World Model for Hidden Patient-State Readout, Reconstruction, and Simulation13 Aug 2026
- 2026UniWorld-Design: From Pixel Generation to Layer-Native Design4 Aug 2026
- 2026STBridge: Shared-Target Alignment for Bridging Understanding and Generation in UMMs19 Jul 2026
- 2026WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modeling3 Jul 2026
- 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling30 Apr 2026
- 2026Advances in GRPO for Generation Models: A Survey21 Feb 2026
- 2026UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models9 Feb 2026
- 2026Mind-Brush: Integrating Agentic Cognitive Search and Reasoning into Image Generation2 Feb 2026
- 2026Think-Then-Generate: Reasoning-Aware Text-to-Image Diffusion with LLM Encoders15 Jan 2026
- 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning30 Oct 2025
- 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 56
- 2025Seeing Before Reasoning: A Unified Framework for Generalizable and Explainable Fake Image Detectioncited by 20
- 2026Think-Then-Generate: Reasoning-Aware Text-to-Image Diffusion with LLM Encoderscited by 15
- 2026Advances in GRPO for Generation Models: A Surveycited by 2
- 2026Advances in GRPO for Generation Models: A Survey21 Feb 2026
- 2026Think-Then-Generate: Reasoning-Aware Text-to-Image Diffusion with LLM Encoders15 Jan 2026
- 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning30 Oct 2025
- 2025Seeing Before Reasoning: A Unified Framework for Generalizable and Explainable Fake Image Detection29 Sep 2025
- 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modelingcited by 5
- 2026WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modelingcited by 0
- 2026HounsWorld: A Multimodal World Model for Hidden Patient-State Readout, Reconstruction, and Simulationcited by 0
- 2026HounsWorld: A Multimodal World Model for Hidden Patient-State Readout, Reconstruction, and Simulation13 Aug 2026
- 2026WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modeling3 Jul 2026
- 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling30 Apr 2026
- 2026UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Modelscited by 7
- 2026Dreaming in Flow: Generative Grounding Feedback for Self-Evolving Unified Multimodal Modelscited by 3
- 2026STBridge: Shared-Target Alignment for Bridging Understanding and Generation in UMMscited by 0
- 2026UniWorld-Design: From Pixel Generation to Layer-Native Designcited by 0
- 2026Editable Visual Designcited by 0
- 2026Dreaming in Flow: Generative Grounding Feedback for Self-Evolving Unified Multimodal Models8 Sep 2026
- 2026Editable Visual Design3 Sep 2026
- 2026UniWorld-Design: From Pixel Generation to Layer-Native Design4 Aug 2026
- 2026STBridge: Shared-Target Alignment for Bridging Understanding and Generation in UMMs19 Jul 2026
- 2026UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models9 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2026Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization
- 2026Unified Text-Image Generation with Weakness-Targeted Post-Training
- 2026Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models
- 2026ReasonUni: Reasoning-Augmented Reconstruction Advances Unified Multimodal Models
- 2025ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
- 2026FiRe: Fine-grained Multimodal Reasoning for Enhanced Image Generation
- 2024VersaT2I: Improving Text-to-Image Models with Versatile Reward
- 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models