mlsanityData to 14 Oct 2026

HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generation

arXiv 2502.12148 ↗First posted 17 Feb 2025cs.CVcited by 18

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The remarkable success of the autoregressive paradigm has made significant advancement in Multimodal Large Language Models (MLLMs), with powerful models like Show-o, Transfusion and Emu3 achieving notable progress in unified image understanding and generation. For the first time, we uncover a common phenomenon: the understanding capabilities of MLLMs are typically stronger than their generative capabilities, with a significant gap between the two. Building on this insight, we propose HermesFlow, a simple yet general framework designed to seamlessly bridge the gap between understanding and generation in MLLMs. Specifically, we take the homologous data as input to curate homologous preference data of both understanding and generation. Through Pair-DPO and self-play iterative optimization, HermesFlow effectively aligns multimodal understanding and generation using homologous preference data. Extensive experiments demonstrate the significant superiority of our approach over prior methods, particularly in narrowing the gap between multimodal understanding and generation. These findings highlight the potential of HermesFlow as a general alignment framework for next-generation multimodal foundation models. Code: https://github.com/Gen-Verse/HermesFlow

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 76 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

54 of 76 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Calibrated Self-Rewarding Vision Language Models23 May
  2. 2023Generative Multimodal Models are In-Context Learners20 Dec
  3. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  4. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  5. 2023JourneyDB: A Benchmark for Generative Image Understanding3 Jul
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  7. 2023Generating Images with Multimodal Language Models26 May

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
35 more
  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  2. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  3. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  4. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  5. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  6. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  7. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  8. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  9. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  10. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  11. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  12. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  13. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  14. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  15. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  16. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  17. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  18. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  19. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  20. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  21. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 129
  22. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  23. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 114
  24. 2024Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Modelscited by 91
  25. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 63
  26. 2024VideoTetris: Towards Compositional Text-to-Video Generationcited by 51
  27. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  28. 2024X-VILA: Cross-Modality Alignment for Large Language Modelcited by 46
  29. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encodingcited by 44
  30. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
  31. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generationcited by 33
  32. 2024Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Taskscited by 32
  33. 2024FATE: Full-head Gaussian Avatar with Textural Editing from Monocular Videocited by 32
  34. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  35. 2024Towards Native Generative Model for 3D Head Avatarcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 8.

18 papers.

Sort:

By area:

  1. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  2. 2025MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Modelscited by 24
  3. 2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learningcited by 24
  4. 2025Generative Universal Verifier as Multimodal Meta-Reasonercited by 17
  5. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  6. 2025PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoningcited by 11
  7. 2025Turning Internal Gap into Self-Improvement: Promoting the Generation-Understanding Unification in MLLMscited by 10
  8. 2025Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forwardcited by 10
  9. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  10. 2026CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generationcited by 6
  11. 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modelingcited by 5
  12. 2026OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibrationcited by 2
6 more
  1. 2025PairUni: Pairwise Training for Unified Multimodal Language Modelscited by 1
  2. 2026Correlation-Weighted Multi-Reward Optimization for Compositional Generationcited by 1
  3. 2026Self-Corrected Image Generation with Explainable Latent Rewardscited by 1
  4. 2026LatentUMM: Dual Latent Alignment for Unified Multimodal Modelscited by 1
  5. 2026Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Modelscited by 0
  6. 2026RSIgen: Reciprocal Self-Improvement for Agentic Visual Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  2. 2024The Revolution of Multimodal Large Language Models: A Survey
  3. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens
  4. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model
  5. 2025On Fairness of Unified Multimodal Large Language Model for Image Generation
  6. 2026Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces
  7. 2026Do Understanding and Generation Fight? A Diagnostic Study of DPO for Unified Multimodal Models
  8. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks