mlsanityData to 7 Oct 2026

Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning

arXiv 2505.03318 ↗First posted 6 May 2025cs.CVcited by 86

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Recent advances in multimodal Reward Models (RMs) have shown significant promise in delivering reward signals to align vision models with human preferences. However, current RMs are generally restricted to providing direct responses or engaging in shallow reasoning processes with limited depth, often leading to inaccurate reward signals. We posit that incorporating explicit long chains of thought (CoT) into the reward reasoning process can significantly strengthen their reliability and robustness. Furthermore, we believe that once RMs internalize CoT reasoning, their direct response accuracy can also be improved through implicit reasoning capabilities. To this end, this paper proposes UnifiedReward-Think, the first unified multimodal CoT-based reward model, capable of multi-dimensional, step-by-step long-chain reasoning for both visual understanding and generation reward tasks. Specifically, we adopt an exploration-driven reinforcement fine-tuning approach to elicit and incentivize the model's latent complex reasoning ability: (1) We first use a small amount of image generation preference data to distill the reasoning process of GPT-4o, which is then used for the model's cold start to learn the format and structure of CoT reasoning. (2) Subsequently, by leveraging the model's prior knowledge and generalization capabilities, we prepare large-scale unified multimodal preference data to elicit the model's reasoning process across various vision tasks. During this phase, correct reasoning outputs are retained for rejection sampling to refine the model (3) while incorrect predicted samples are finally used for Group Relative Policy Optimization (GRPO) based reinforcement fine-tuning, enabling the model to explore diverse reasoning paths and optimize for correct and robust solutions. Extensive experiments across various vision reward tasks demonstrate the superiority of our model.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 44 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

38 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Visual-RFT: Visual Reinforcement Fine-Tuning3 Mar
  2. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model21 Jan

Other references · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  10. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  12. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
24 more
  1. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  4. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  5. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  6. 2025Improving Video Generation with Human Feedbackcited by 267
  7. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  8. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  9. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  10. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  11. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
  12. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  13. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  14. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 117
  15. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 102
  16. 2025Q-Insight: Understanding Image Quality via Visual Reinforcement Learningcited by 93
  17. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  18. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 81
  19. 2024o1-Coder: an o1 Replication for Codingcited by 77
  20. 2024LiFT: Leveraging Human Feedback for Text-to-Video Model Alignmentcited by 51
  21. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 38
  22. 2024EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluationcited by 37
  23. 2024CodeDPO: Aligning Code Models with Self Generated and Verified Source Codecited by 32
  24. 2024Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 39; 2026: 47.

86 papers.

Sort:

By area:

  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  2. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  3. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 88
  4. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 64
  5. 2025RewardDance: Reward Scaling in Visual Generationcited by 63
  6. 2025EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modelingcited by 61
  7. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 47
  8. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  9. 2025Does FLUX Already Know How to Perform Physically Plausible Image Composition?cited by 30
  10. 2025VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learningcited by 26
  11. 2025UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generationcited by 23
  12. 2025UniREditBench: A Unified Reasoning-based Image Editing Benchmarkcited by 23
38 more
  1. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  2. 2025Think-J: Learning to Think for Generative LLM-as-a-Judgecited by 20
  3. 2025Reasoning as Representation: Rethinking Visual Reinforcement Learning in Image Quality Assessmentcited by 18
  4. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  5. 2026DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editingcited by 14
  6. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 13
  7. 2025IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarioscited by 12
  8. 2025VRPRM: Process Reward Modeling via Visual Reasoningcited by 11
  9. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 11
  10. 2026Unified Personalized Reward Model for Vision Generationcited by 11
  11. 2026GEditBench v2: A Human-Aligned Benchmark for General Image Editingcited by 11
  12. 2025Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusabilitycited by 10
  13. 2026PhyCritic: Multimodal Critic Models for Physical AIcited by 9
  14. 2025Reinforcement Learning for Large Model: A Surveycited by 8
  15. 2025Are We Ready for RL in Text-to-3D Generation? A Progressive Investigationcited by 8
  16. 2026AI Can Learn Scientific Tastecited by 8
  17. 2026Leveraging Verifier-Based Reinforcement Learning in Image Editingcited by 8
  18. 2026Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteriacited by 8
  19. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  20. 2025AutoPP: Towards Automated Product Poster Generation and Optimizationcited by 7
  21. 2026GDRO: Group-level Reward Post-training Suitable for Diffusion Modelscited by 7
  22. 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesiscited by 7
  23. 2025Reason-SVG: Enhancing Structured Reasoning for Vector Graphics Generation with Reinforcement Learningcited by 6
  24. 2025ImageDoctor: Diagnosing Text-to-Image Generation via Grounded Image Reasoningcited by 6
  25. 2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understandingcited by 5
  26. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  27. 2026Beyond VLM-Based Rewards: Diffusion-Native Latent Reward Modelingcited by 5
  28. 2026PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignmentcited by 5
  29. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learningcited by 5
  30. 2025Training Multi-Image Vision Agents via End2End Reinforcement Learningcited by 4
  31. 2026Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicinecited by 4
  32. 2026SpatialReward: Verifiable Spatial Reward Modeling for Fine-Grained Spatial Consistency in Text-to-Image Generationcited by 4
  33. 2025T2I-Eval-R1: Reinforcement Learning-Driven Reasoning for Interpretable Text-to-Image Evaluationcited by 3
  34. 2025Personalized Reward Modeling for Text-to-Image Generationcited by 3
  35. 2026EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessmentcited by 3
  36. 2026Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Modelscited by 3
  37. 2026Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generationcited by 3
  38. 2026Discriminative Perception via Anchored Description for Reasoning Segmentationcited by 3

50 most cited of 86 All 86 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards
  2. 2026Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought
  3. 2025Latent Chain-of-Thought for Visual Reasoning
  4. 2025Benchmarking Multimodal CoT Reward Model Stepwise by Visual Program
  5. 2026Disentangling Perception and Reasoning in Multimodal LLMs via Reward Design
  6. 2025ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning
  7. 2026Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models
  8. 2026Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning