mlsanityData to 14 Oct 2026

R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning

arXiv 2505.02835 ↗First posted 5 May 2025cs.CV cs.CLcited by 65

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Multimodal Reward Models (MRMs) play a crucial role in enhancing the performance of Multimodal Large Language Models (MLLMs). While recent advancements have primarily focused on improving the model structure and training data of MRMs, there has been limited exploration into the effectiveness of long-term reasoning capabilities for reward modeling and how to activate these capabilities in MRMs. In this paper, we explore how Reinforcement Learning (RL) can be used to improve reward modeling. Specifically, we reformulate the reward modeling problem as a rule-based RL task. However, we observe that directly applying existing RL algorithms, such as Reinforce++, to reward modeling often leads to training instability or even collapse due to the inherent limitations of these algorithms. To address this issue, we propose the StableReinforce algorithm, which refines the training loss, advantage estimation strategy, and reward design of existing RL methods. These refinements result in more stable training dynamics and superior performance. To facilitate MRM training, we collect 200K preference data from diverse datasets. Our reward model, R1-Reward, trained using the StableReinforce algorithm on this dataset, significantly improves performance on multimodal reward modeling benchmarks. Compared to previous SOTA models, R1-Reward achieves a 8.4% improvement on the VL Reward-Bench and a 14.3% improvement on the Multimodal Reward Bench. Moreover, with more inference compute, R1-Reward's performance is further enhanced, highlighting the potential of RL algorithms in optimizing MRMs.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 64 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

56 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
  2. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL10 Mar
  3. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models9 Mar

Other references · 53

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  10. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
38 more
  1. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  2. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  4. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  5. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  6. 2025Process Reinforcement through Implicit Rewardscited by 504
  7. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  8. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  9. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  10. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  11. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  12. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  13. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  14. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  15. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  16. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  17. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 166
  18. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 166
  19. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  20. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
  21. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 130
  22. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  23. 2025R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learningcited by 100
  24. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 86
  25. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
  26. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 76
  27. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  28. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 74
  29. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 74
  30. 2019A Survey of Deep Reinforcement Learning in Video Gamescited by 73
  31. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  32. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 71
  33. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  34. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 44
  35. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  36. 2024Debiasing Multimodal Large Language Models via Penalization of Language Priorscited by 37
  37. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 34
  38. 2024Benchmarking Large Multimodal Models against Common Corruptionscited by 24

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 32; 2026: 33.

65 papers.

Sort:

By area:

  1. 2025Thyme: Think Beyond Imagescited by 167
  2. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
  3. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  4. 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
  5. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 50
  6. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 35
  7. 2025SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learningcited by 31
  8. 2025Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearchcited by 27
  9. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmarkcited by 21
  10. 2025Audio-Thinker: Guiding Audio Language Model When and How to Think via Reinforcement Learningcited by 19
  11. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  12. 2025TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMscited by 15
38 more
  1. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 15
  2. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 14
  3. 2025ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoningcited by 13
  4. 2025VRPRM: Process Reward Modeling via Visual Reasoningcited by 11
  5. 2026PhyCritic: Multimodal Critic Models for Physical AIcited by 10
  6. 2025VACoT: Rethinking Visual Data Augmentation with VLMscited by 9
  7. 2026Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMscited by 9
  8. 2025Reason-SVG: Enhancing Structured Reasoning for Vector Graphics Generation with Reinforcement Learningcited by 8
  9. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  10. 2025TIM-PRM: Verifying multimodal reasoning with Tool-Integrated PRMcited by 8
  11. 2025GRAIL:Learning to Interact with Large Knowledge Graphs for Retrieval Augmented Reasoningcited by 7
  12. 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesiscited by 7
  13. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  14. 2025Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Modelscited by 6
  15. 2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understandingcited by 6
  16. 2025Rethinking Reward Signals in Video GRPO: When Scores Become Targetscited by 6
  17. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learningcited by 5
  18. 2026StaRPO: Stability-Augmented Reinforcement Policy Optimizationcited by 5
  19. 2026Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVRcited by 5
  20. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  21. 2025Divide-Fuse-Conquer: Eliciting "Aha Moments" in Multi-Scenario Gamescited by 4
  22. 2025Training Multi-Image Vision Agents via End2End Reinforcement Learningcited by 4
  23. 2026Enhancing Multi-Modal LLMs Reasoning via Difficulty-Aware Group Normalizationcited by 4
  24. 2026Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Modelscited by 4
  25. 2026Improving Vision-language Models with Perception-centric Process Reward Modelscited by 4
  26. 2026OPERA: An Agent for Image Restoration with End-to-End Joint Planning-Execution Optimizationcited by 4
  27. 2026Thinking with Frames: Generative Video Distortion Evaluation via Frame Reward Modelcited by 3
  28. 2026Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Modelscited by 3
  29. 2026Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generationcited by 3
  30. 2026Agentic Critical Trainingcited by 3
  31. 2026See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflectioncited by 3
  32. 2026DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verificationcited by 3
  33. 2025Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Modelscited by 2
  34. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  35. 2026CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteriacited by 2
  36. 2026See More, Think Deeper: Query-Expanded Visual Evidence and Answer-Clue Guided Reflection for Long Video Understandingcited by 2
  37. 2025Enhancing Agentic Textual Graph Retrieval with Synthetic Stepwise Supervisioncited by 1
  38. 2025MARS-Sep: Multimodal-Aligned Reinforced Sound Separationcited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Models
  2. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models
  3. 2024Reward-Robust RLHF in LLMs
  4. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  5. 2025RM-R1: Reward Modeling as Reasoning
  6. 2025Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment
  7. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  8. 2025Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling