Data to 7 Oct 2026

Reward Reasoning Model

arXiv 2505.14674 ↗First posted 20 May 2025cs.CLcited by 47

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more

LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Reward models play a critical role in guiding large language models toward outputs that align with human expectations. However, an open challenge remains in effectively utilizing test-time compute to enhance reward model performance. In this work, we introduce Reward Reasoning Models (RRMs), which are specifically designed to execute a deliberate reasoning process before generating final rewards. Through chain-of-thought reasoning, RRMs leverage additional test-time compute for complex queries where appropriate rewards are not immediately apparent. To develop RRMs, we implement a reinforcement learning framework that fosters self-evolved reward reasoning capabilities without requiring explicit reasoning traces as training data. Experimental results demonstrate that RRMs achieve superior performance on reward modeling benchmarks across diverse domains. Notably, we show that RRMs can adaptively exploit test-time compute to further improve reward accuracy. The pretrained reward reasoning models are available at https://huggingface.co/Reward-Reasoning.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 80 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

37 of 80 references.

Closest ancestors · 10

Same direction, newest first

Other references · 27

2023GPT-4 Technical Reportcited by 18,132
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2021Training Verifiers to Solve Math Word Problemscited by 9,743
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
2020Scaling Laws for Neural Language Modelscited by 6,411
2024Qwen2.5 Technical Reportcited by 4,713
2024GPT-4o System Cardcited by 4,467
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2024Qwen2 Technical Reportcited by 2,076
2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
2024A Survey on LLM-as-a-Judgecited by 1,384
2024OpenAI o1 System Cardcited by 1,220

15 more

2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
2024Generative Reward Modelscited by 148
2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
2025JudgeLRM: Large Reasoning Models as a Judgecited by 76
2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
2024Self-Taught Evaluatorscited by 70
2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
2025Atla Selene Mini: A General Purpose Evaluation Modelcited by 27
2025PairJudge RM: Perform Best-of-N Sampling with Knockout Tournamentcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 24.

47 papers.

Sort:

By area:

2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
2025GRAM: A Generative Foundation Reward Model for Reward Generalizationcited by 25
2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
2025Learning to Refine: Self-Refinement of Parallel Reasoning in LLMscited by 12
2025RecGPT Technical Reportcited by 11
2025VRPRM: Process Reward Modeling via Visual Reasoningcited by 11
2026Advancing General-Purpose Reasoning Models with Modular Gradient Surgerycited by 9

35 more

2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Modelscited by 9
2026LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewardscited by 9
2026AI Can Learn Scientific Tastecited by 8
2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
2025Think Twice: Branch-and-Rethink Reasoning Reward Modelcited by 7
2026Reward Modeling from Natural Language Human Feedbackcited by 7
2026WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agentscited by 7
2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
2025HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Applicationcited by 5
2025The Era of Agentic Organization: Learning to Organize with Language Modelscited by 5
2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
2025Libra: Assessing and Improving Reward Model by Learning to Thinkcited by 4
2026Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verificationcited by 4
2026Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Modelscited by 4
2025DEL-ToM: Inference-Time Scaling for Theory-of-Mind Reasoning via Dynamic Epistemic Logiccited by 3
2025Brittleness and Promise: Knowledge Graph Based Reward Modeling for Diagnostic Reasoningcited by 3
2025Best-of-∞ -- Asymptotic Performance of Test-Time LLM Ensemblingcited by 3
2026WildReward: Learning Reward Models from In-the-Wild Human Interactionscited by 3
2025PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learningcited by 2
2025Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignmentcited by 2
2025OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learningcited by 2
2026Reward Prediction with Factorized World Statescited by 2
2026CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteriacited by 1
2026Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debatecited by 1
2026Learning Ordinal Probabilistic Reward from Preferencescited by 1
2026Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Modelscited by 1
2026R-Align: Enhancing Generative Reward Models through Rationale-Centric Meta-Judgingcited by 0
2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 0
2026Biological Reasoning-Informed Regression for Interpretable Regulatory DNA Activity Predictioncited by 0
2026Counsel: A Meta-Evaluation Dataset for Agentic Taskscited by 0
2026Deep Interaction: An Efficient Human-AI Interaction Method for Large Reasoning Modelscited by 0
2026TAPAS: Throughput-adaptive Perception for Autonomous Systemscited by 0
2026VlogReward: Learning Multi-Dimensional Evaluation for Vlog Editingcited by 0
2026From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMscited by 0
2026Training LLM Judges from Language Feedback via Position-Selective Self-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way