mlsanityData to 14 Oct 2026

Unified Reward Model for Multimodal Understanding and Generation

arXiv 2503.05236 ↗First posted 7 Mar 2025cs.CVcited by 199

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Recent advances in human preference alignment have significantly improved multimodal generation and understanding. A key approach is to train reward models that provide supervision signals for preference optimization. However, existing reward models are often task-specific, limiting their adaptability across diverse visual applications. We also argue that a reward model that jointly learning to assess multiple vision tasks may foster a synergistic effect, where improved image understanding enhances image generation assessment, and refined image evaluation benefits video assessment through better frame analysis. To this end, this paper proposes UnifiedReward, the first unified reward model for multimodal understanding and generation assessment. It supports both pairwise ranking and pointwise scoring, providing effective reward signals for vision model preference alignment. Specifically, (1) we first train UnifiedReward on our constructed large-scale human preference dataset, which covers both image and video generation/understanding tasks. (2) Then, we leverage it to automatically construct high-quality pairwise preference data from vision models by progressively filtering their outputs through our two-stage strategy, i.e., pair ranking and point sifting. (3) Finally, we use these data to align vision models with human preferences via Direct Preference Optimization (DPO). Experimental results show that jointly learning to assess diverse visual tasks yields substantial mutual benefits. We further apply our pipeline to both vision understanding and generation, achieving consistent improvements across each domain.

Key facts. Cited by 199 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 63 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

53 of 63 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025Temporal Preference Optimization for Long-Form Video Understanding23 Jan
  2. 2025Improving Video Generation with Human Feedback23 Jan
  3. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model21 Jan
  4. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation30 Dec
  5. 2024OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimization19 Dec
  6. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generation18 Dec
  7. 2024LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment6 Dec
  8. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback3 Dec
  9. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models26 Nov
  10. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design8 Oct
  11. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward1 Apr
  12. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation15 Feb
7 more
  1. 2024Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback6 Feb
  2. 2023Rich Human Feedback for Text-to-Image Generation15 Dec
  3. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  4. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  6. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  7. 2023Aligning Text-to-Image Models using Human Feedback23 Feb

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  11. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  12. 2019Towards VQA Models That Can Readcited by 1,718
22 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  5. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  6. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  7. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  8. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  9. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  10. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  11. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  12. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 260
  13. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
  14. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  15. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  16. 2024Learning Multi-dimensional Human Preference for Text-to-Image Generationcited by 106
  17. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 82
  18. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  19. 2024EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluationcited by 39
  20. 2020Document Visual Question Answering Challenge 2020cited by 20
  21. 2024Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluationcited by 7
  22. 2023MSVD-Indonesian: A Benchmark for Multimodal Video-Text Tasks in Indonesiancited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 80; 2026: 119.

199 papers.

Sort:

By area:

  1. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  2. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 230
  3. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 178
  4. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  5. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 92
  6. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 90
  7. 2025Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillationcited by 86
  8. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 79
  9. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 69
  10. 2025EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modelingcited by 64
  11. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 61
  12. 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 54
38 more
  1. 2025Distribution Matching Distillation Meets Reinforcement Learningcited by 50
  2. 2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clippingcited by 49
  3. 2025Interleaving Reasoning for Better Text-to-Image Generationcited by 41
  4. 2025VideoScore2: Think before You Score in Generative Video Evaluationcited by 40
  5. 2026DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Modelscited by 40
  6. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 38
  7. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 36
  8. 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 34
  9. 2026Flow-OPD: On-Policy Distillation for Flow Matching Modelscited by 34
  10. 2025Does FLUX Already Know How to Perform Physically Plausible Image Composition?cited by 32
  11. 2025Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPOcited by 31
  12. 2025UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generationcited by 25
  13. 2025UniREditBench: A Unified Reasoning-based Image Editing Benchmarkcited by 25
  14. 2025Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoningcited by 23
  15. 2025FingER: Content Aware Fine-grained Evaluation with Reasoning for AI-Generated Videoscited by 22
  16. 2025AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learningcited by 22
  17. 2025Fine-Grained GRPO for Precise Preference Alignment in Flow Modelscited by 21
  18. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignmentcited by 21
  19. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Imagecited by 20
  20. 2025Follow-Your-Preference: Towards Preference-Aligned Image Inpaintingcited by 18
  21. 2025DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPOcited by 17
  22. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  23. 2025Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Modelscited by 16
  24. 2025Video Generation Models Are Good Latent Reward Modelscited by 16
  25. 2025VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimizationcited by 16
  26. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  27. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 15
  28. 2026TextPecker: Rewarding Structural Anomaly Quantification for Enhancing Visual Text Renderingcited by 15
  29. 2025Reinforcing Diffusion Models by Direct Group Preference Optimizationcited by 14
  30. 2026RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Timecited by 14
  31. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
  32. 2025IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarioscited by 13
  33. 2025ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoningcited by 13
  34. 2026Unified Personalized Reward Model for Vision Generationcited by 13
  35. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 12
  36. 2025Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Modelscited by 12
  37. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 12
  38. 2025The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generationcited by 12

50 most cited of 199 All 199 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models
  2. 2026Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models
  3. 2024RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data
  4. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models
  5. 2026Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Models
  6. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models
  7. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
  8. 2025OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learning