mlsanityData to 7 Oct 2026

UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation

arXiv 2505.14682 ↗First posted 20 May 2025cs.CVcited by 32

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce UniGen, a unified multimodal large language model (MLLM) capable of image understanding and generation. We study the full training pipeline of UniGen from a data-centric perspective, including multi-stage pre-training, supervised fine-tuning, and direct preference optimization. More importantly, we propose a new Chain-of-Thought Verification (CoT-V) strategy for test-time scaling, which significantly boosts UniGen's image generation quality using a simple Best-of-N test-time strategy. Specifically, CoT-V enables UniGen to act as both image generator and verifier at test time, assessing the semantic alignment between a text prompt and its generated image in a step-by-step CoT manner. Trained entirely on open-source datasets across all stages, UniGen achieves state-of-the-art performance on a range of image understanding and generation benchmarks, with a final score of 0.78 on GenEval and 85.19 on DPG-Bench. Through extensive ablation studies, our work provides actionable insights and addresses key challenges in the full life cycle of building unified MLLMs, contributing meaningful directions to the future research.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 88 of its 105 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

88 of 105 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generation17 Feb
  2. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance9 Dec
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  4. 2024LLaVA-Critic: Learning to Evaluate Multimodal Models3 Oct
  5. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning30 Sep
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  7. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  8. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  9. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  12. 2023JourneyDB: A Benchmark for Generative Image Understanding3 Jul

Other references · 76

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Segment Anythingcited by 8,158
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2025Qwen2.5-VL Technical Reportcited by 5,418
  11. 2024Qwen2.5 Technical Reportcited by 4,713
  12. 2017Neural Discrete Representation Learningcited by 4,650
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  12. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  13. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  14. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  15. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  16. 2016A Diagram Is Worth A Dozen Imagescited by 793
  17. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  18. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  19. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  20. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  21. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  22. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  23. 2021ImageNet-21K Pretraining for the Massescited by 607
  24. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  25. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  26. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  27. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  28. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  29. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  30. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
  31. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  32. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  33. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  34. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  35. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 294
  36. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  37. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  38. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 19.

32 papers.

Sort:

By area:

  1. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  2. 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 50
  3. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 48
  4. 2025AToken: A Unified Tokenizer for Visioncited by 30
  5. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 23
  6. 2025Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generationcited by 14
  7. 2025Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Controlcited by 10
  8. 2025UniGen-1.5: Enhancing Image Generation and Editing through Reward Unification in Reinforcement Learningcited by 8
  9. 2026From Scale to Speed: Adaptive Test-Time Scaling for Image Editingcited by 7
  10. 2025DeRA: Decoupled Representation Alignment for Video Tokenizationcited by 6
  11. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 6
  12. 2026Meta-TTRL: A Metacognitive Framework for Self-Improving Test-Time Reinforcement Learning in Unified Multimodal Modelscited by 4
20 more
  1. 2026OmniGen-AR: AutoRegressive Any-to-Image Generationcited by 4
  2. 2025POET: Preference Optimization for Enhanced Text-to-Image Generationcited by 3
  3. 2025UniHetero: Could Generation Enhance Understanding for Vision-Language-Model at Large Data Scale?cited by 3
  4. 2026Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unificationcited by 3
  5. 2025What Happens Next? Next Scene Prediction with a Unified Video Modelcited by 2
  6. 2025Omni-Weather: A Unified Multimodal Model for Weather Radar Understanding and Generationcited by 2
  7. 2026Unsafe by Reciprocity: How Generation-Understanding Coupling Undermines Safety in Unified Multimodal Modelscited by 2
  8. 2026Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generationcited by 2
  9. 2025Rethinking Prompt Design for Inference-time Scaling in Text-to-Visual Generationcited by 1
  10. 2026Unified Text-Image Generation with Weakness-Targeted Post-Trainingcited by 1
  11. 2026Large Language Models are Universal Reasoners for Visual Generationcited by 1
  12. 2026Verification Mirage: Mapping the Reliability Boundary of Self-Verification in Medical VQAcited by 1
  13. 2026ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representationscited by 1
  14. 2026Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoningcited by 1
  15. 2026Uni-Synergy: Bridging Understanding and Generation for Personalized Reasoning via Co-operative Reinforcement Learningcited by 0
  16. 2026Test-Time Scaling in Multimodal Foundation Models: A Comprehensive Survey of Generation and Reasoningcited by 0
  17. 2026STBridge: Shared-Target Alignment for Bridging Understanding and Generation in UMMscited by 0
  18. 2026Agentic Visual Generation: From Generative Models to Agentic Controlcited by 0
  19. 2026Fysiverse-3D-Vision Technical Report: Generating Executable 3D Worlds from Images through Unified Spatial Reasoningcited by 0
  20. 2026PreviewDiff: Multimodal Critic-Guided Search over Diffusion Latentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation
  2. 2026UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark
  3. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  4. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
  5. 2026UEval: A Benchmark for Unified Multimodal Generation
  6. 2026UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling
  7. 2025OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models
  8. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding