# Reward Reasoning Model arXiv: https://arxiv.org/abs/2505.14674 · HTML: https://mlsanity.com/paper/2505.14674/ ## Key facts - Cited by 47 papers in the map (data through 14 Oct 2026). - Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. - Builds on 37 of its 80 references that are in the map. - First posted on arXiv 20 May 2025. ## Placed in - LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/) - LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/) ## Builds on (most cited first, 30 of 37) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024) - [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024) - [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022) - [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022) - [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025) - [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024) - [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024) - [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024) - [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024) - [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024) - [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024) - [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024) - [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024) - [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024) - [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://mlsanity.com/paper/2501.07301/) (2025) - [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024) - [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025) - [Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge](https://mlsanity.com/paper/2407.19594/) (2024) - [Secrets of RLHF in Large Language Models Part II: Reward Modeling](https://mlsanity.com/paper/2401.06080/) (2024) - [RM-R1: Reward Modeling as Reasoning](https://mlsanity.com/paper/2505.02387/) (2025) - [Generative Reward Models](https://mlsanity.com/paper/2410.12832/) (2024) - [HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM](https://mlsanity.com/paper/2311.09528/) (2023) - [Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models](https://mlsanity.com/paper/2502.17387/) (2025) - [On Designing Effective RL Reward at Training Time for LLM Reasoning](https://mlsanity.com/paper/2410.15115/) (2024) ## Built on by (most cited first, 30 of 47) - [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025) - [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025) - [OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment](https://mlsanity.com/paper/2510.07743/) (2025) - [Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle](https://mlsanity.com/paper/2509.16679/) (2025) - [Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling](https://mlsanity.com/paper/2510.17314/) (2025) - [GRAM: A Generative Foundation Reward Model for Reward Generalization](https://mlsanity.com/paper/2506.14175/) (2025) - [Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning](https://mlsanity.com/paper/2510.23038/) (2025) - [Pre-Trained Policy Discriminators are General Reward Models](https://mlsanity.com/paper/2507.05197/) (2025) - [Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs](https://mlsanity.com/paper/2509.00084/) (2025) - [RecGPT Technical Report](https://mlsanity.com/paper/2507.22879/) (2025) - [VRPRM: Process Reward Modeling via Visual Reasoning](https://mlsanity.com/paper/2508.03556/) (2025) - [Reward Modeling from Natural Language Human Feedback](https://mlsanity.com/paper/2601.07349/) (2026) - [Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models](https://mlsanity.com/paper/2602.04649/) (2026) - [AI Can Learn Scientific Taste](https://mlsanity.com/paper/2603.14473/) (2026) - [LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards](https://mlsanity.com/paper/2602.05758/) (2026) - [Advancing General-Purpose Reasoning Models with Modular Gradient Surgery](https://mlsanity.com/paper/2602.02301/) (2026) - [Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey](https://mlsanity.com/paper/2510.01925/) (2025) - [Think Twice: Branch-and-Rethink Reasoning Reward Model](https://mlsanity.com/paper/2510.23596/) (2025) - [The Era of Agentic Organization: Learning to Organize with Language Models](https://mlsanity.com/paper/2510.26658/) (2025) - [WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agents](https://mlsanity.com/paper/2601.21872/) (2026) - [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026) - [S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models](https://mlsanity.com/paper/2509.22099/) (2025) - [HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application](https://mlsanity.com/paper/2510.19631/) (2025) - [Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verification](https://mlsanity.com/paper/2601.22642/) (2026) - [Libra: Assessing and Improving Reward Model by Learning to Think](https://mlsanity.com/paper/2507.21645/) (2025) - [Brittleness and Promise: Knowledge Graph Based Reward Modeling for Diagnostic Reasoning](https://mlsanity.com/paper/2509.18316/) (2025) - [Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Models](https://mlsanity.com/paper/2603.16253/) (2026) - [DEL-ToM: Inference-Time Scaling for Theory-of-Mind Reasoning via Dynamic Epistemic Logic](https://mlsanity.com/paper/2505.17348/) (2025) - [Best-of-∞ -- Asymptotic Performance of Test-Time LLM Ensembling](https://mlsanity.com/paper/2509.21091/) (2025) - [R-Align: Enhancing Generative Reward Models through Rationale-Centric Meta-Judging](https://mlsanity.com/paper/2602.06763/) (2026) ## Abstract (arXiv) Reward models play a critical role in guiding large language models toward outputs that align with human expectations. However, an open challenge remains in effectively utilizing test-time compute to enhance reward model performance. In this work, we introduce Reward Reasoning Models (RRMs), which are specifically designed to execute a deliberate reasoning process before generating final rewards. Through chain-of-thought reasoning, RRMs leverage additional test-time compute for complex queries where appropriate rewards are not immediately apparent. To develop RRMs, we implement a reinforcement learning framework that fosters self-evolved reward reasoning capabilities without requiring explicit reasoning traces as training data. Experimental results demonstrate that RRMs achieve superior performance on reward modeling benchmarks across diverse domains. Notably, we show that RRMs can adaptively exploit test-time compute to further improve reward accuracy. The pretrained reward reasoning models are available at https://huggingface.co/Reward-Reasoning.