mlsanityData to 7 Oct 2026

RM-R1: Reward Modeling as Reasoning

arXiv 2505.02387 ↗First posted 5 May 2025cs.CL cs.AI cs.LGcited by 149

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reward modeling is essential for aligning large language models with human preferences through reinforcement learning. To provide accurate reward signals, a reward model (RM) should stimulate deep thinking and conduct interpretable reasoning before assigning a score or a judgment. Inspired by recent advances of long chain-of-thought on reasoning-intensive tasks, we hypothesize and validate that integrating reasoning into reward modeling significantly enhances RM's interpretability and performance. We introduce a new class of generative reward models, Reasoning Reward Models (ReasRMs), which formulate reward modeling as a reasoning task. We propose a reasoning-oriented training pipeline and train a family of ReasRMs, RM-R1. RM-R1 features a chain-of-rubrics (CoR) mechanism -- self-generating sample-level chat rubrics or math/code solutions, and evaluating candidate responses against them. The training of RM-R1 consists of two key stages: (1) distillation of high-quality reasoning chains and (2) reinforcement learning with verifiable rewards. Empirically, our models achieve superior performance across three reward model benchmarks on average, outperforming much larger open-weight models (e.g., INF-ORM-Llama3.1-70B) and proprietary ones (e.g., GPT-4o) by up to 4.9%. Beyond final performance, we perform thorough analyses to understand the key ingredients of successful ReasRM training.

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 47 of its 69 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

47 of 69 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  2. 2025Process Reinforcement through Implicit Rewards3 Feb
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2024Self-Generated Critiques Boost Reward Modeling for Language Models25 Nov
  5. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  6. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  7. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  8. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  10. 2023Let's Verify Step by Step31 May
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
1 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2024GPT-4o System Cardcited by 4,467
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
22 more
  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  3. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  4. 2024InternLM2 Technical Reportcited by 391
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  6. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  7. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  8. 2024Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoderscited by 213
  9. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  10. 2021Does Knowledge Distillation Really Work?cited by 187
  11. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  12. 2025ToRL: Scaling Tool-Integrated RLcited by 166
  13. 2024Nemotron-4 340B Technical Reportcited by 143
  14. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  15. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 76
  16. 2024Self-Taught Evaluatorscited by 70
  17. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
  18. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  19. 2024LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedbackcited by 40
  20. 2024DeLLMa: Decision Making Under Uncertainty with Large Language Modelscited by 20
  21. 2023MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answeringcited by 12
  22. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Makingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 77; 2026: 72.

149 papers.

Sort:

By area:

  1. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  5. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  6. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  7. 2025RLPR: Extrapolating RLVR to General Domains without Verifierscited by 83
  8. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  9. 2025Scaling Test-time Compute for LLM Agentscited by 59
  10. 2025Reward Reasoning Modelcited by 47
  11. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 47
  12. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 46
38 more
  1. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  2. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  3. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  4. 2025SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learningcited by 31
  5. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  6. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
  7. 2025StepWiser: Stepwise Generative Judges for Wiser Reasoningcited by 28
  8. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  9. 2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 27
  10. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  11. 2025GRAM: A Generative Foundation Reward Model for Reward Generalizationcited by 25
  12. 2025OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critiquecited by 24
  13. 2025Time-R1: Towards Comprehensive Temporal Reasoning in LLMscited by 23
  14. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  15. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  16. 2025Think-J: Learning to Think for Generative LLM-as-a-Judgecited by 20
  17. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  18. 2026WebGym: Scaling Training Environments for Visual Web Agents with Realistic Taskscited by 19
  19. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 18
  20. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 18
  21. 2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Taskscited by 16
  22. 2025xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learningcited by 16
  23. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
  24. 2025Training LLMs for EHR-Based Reasoning Tasks via Reinforcement Learningcited by 14
  25. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  26. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoningcited by 14
  27. 2025ToMAP: Training Opponent-Aware LLM Persuaders with Theory of Mindcited by 12
  28. 2025Learning to Refine: Self-Refinement of Parallel Reasoning in LLMscited by 12
  29. 2026AgentV-RL: Scaling Reward Modeling with Agentic Verifiercited by 12
  30. 2025CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewardscited by 11
  31. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11
  32. 2025VRPRM: Process Reward Modeling via Visual Reasoningcited by 11
  33. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  34. 2026Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Trainingcited by 11
  35. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  36. 2025Reasoning-Enhanced Large Language Models for Molecular Property Predictioncited by 9
  37. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Modelscited by 9
  38. 2026LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewardscited by 9

50 most cited of 149 All 149 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
  2. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
  3. 2025Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning
  4. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  5. 2026Learning Latent Reasoning Traces for Scalar Reward Models End-to-End
  6. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
  7. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning
  8. 2025Reinforcement Learning with Rubric Anchors