mlsanityData to 14 Oct 2026

Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey

arXiv 2510.01925 ↗First posted 2 Oct 2025cs.CLcited by 8

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reward models (RMs) play a critical role in enhancing the reasoning performance of LLMs. For example, they can provide training signals to finetune LLMs during reinforcement learning (RL) and help select the best answer from multiple candidates during inference. In this paper, we provide a systematic introduction to RMs, along with a comprehensive survey of their applications in LLM reasoning. We first review fundamental concepts of RMs, including their architectures, training methodologies, and evaluation techniques. Then, we explore their key applications: (1) guiding generation and selecting optimal outputs during LLM inference, (2) facilitating data synthesis and iterative self-improvement for LLMs, and (3) providing training signals in RL-based finetuning. Finally, we discuss critical open questions regarding the selection, generalization, evaluation, and enhancement of RMs, based on existing research and our own empirical findings. Our analysis aims to provide actionable insights for the effective deployment and advancement of RMs for LLM reasoning.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 262 of its 373 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

262 of 373 references.

Closest ancestors · 77

Same direction, newest first

  1. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning30 Sep
  2. 2025The Majority is not always right: RL training for solution aggregation8 Sep
  3. 2025ReCode: Reinforcing Code Generation with Reasoning-Process Rewards7 Aug
  4. 2025Libra: Assessing and Improving Reward Model by Learning to Think29 Jul
  5. 2025CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoning21 Jul
  6. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  7. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs23 Jun
  8. 2025DuaShepherd: Integrating Stepwise Correctness and Potential Rewards for Mathematical Reasoning21 Jun
  9. 2025SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling18 Jun
  10. 2025Magistral12 Jun
  11. 2025Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Models11 Jun
  12. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback3 Jun
38 more
  1. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models29 May
  2. 2025Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns29 May
  3. 2025Reward Reasoning Model20 May
  4. 2025Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs16 May
  5. 2025Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers7 May
  6. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards5 May
  7. 2025RM-R1: Reward Modeling as Reasoning5 May
  8. 2025Process Reward Models That Think23 Apr
  9. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  10. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning21 Apr
  11. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  12. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use7 Apr
  13. 2025Sample, Don't Search: Rethinking Test-Time Alignment for Language Models4 Apr
  14. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  15. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  16. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning1 Apr
  17. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  18. 2025R-PRM: Reasoning-Driven Process Reward Modeling27 Mar
  19. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  20. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  21. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation17 Mar
  22. 2025Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models16 Mar
  23. 2025MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modeling15 Mar
  24. 2025Better Process Supervision with Bi-directional Rewarding Signals6 Mar
  25. 2025Process-based Self-Rewarding Language Models5 Mar
  26. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems26 Feb
  27. 2025AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence19 Feb
  28. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning18 Feb
  29. 2025AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verification17 Feb
  30. 2025Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective14 Feb
  31. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning10 Feb
  32. 2025VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data10 Feb
  33. 2025Process Reinforcement through Implicit Rewards3 Feb
  34. 2025Coarse-to-Fine Process Reward Modeling for Mathematical Reasoning23 Jan
  35. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  36. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  37. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models6 Jan
  38. 2025Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search2 Jan

50 newest of 77 All 77 (JSON)

Other references · 185

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  12. 2016Concrete Problems in AI Safetycited by 1,642
38 more
  1. 2024A Survey on LLM-as-a-Judgecited by 1,423
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  3. 2024OpenAI o1 System Cardcited by 1,244
  4. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  5. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  8. 2023Teaching Large Language Models to Self-Debugcited by 988
  9. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  10. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  11. 2025Group Sequence Policy Optimizationcited by 835
  12. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  13. 2024Self-Rewarding Language Modelscited by 681
  14. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  15. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  16. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  17. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  18. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  19. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  20. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  21. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  22. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  23. 2024InternLM2 Technical Reportcited by 397
  24. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  25. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  26. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  27. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  28. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  29. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  30. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  31. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  32. 2025Improving Video Generation with Human Feedbackcited by 284
  33. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  34. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  35. 2024Iterative Reasoning Preference Optimizationcited by 246
  36. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  37. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  38. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236

50 most cited of 185 All 185 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  2. 2026On the Power of (Approximate) Reward Models for Inference-Time Scalingcited by 7
  3. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  4. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  5. 2026Process Advantage Signal Shaping: A Paradigm-Agnostic Middleware for Process-Supervised RL in LLM Reasonerscited by 2
  6. 2026EMAD: Evidence-Centric Grounded Multimodal Diagnosis for Alzheimer's Diseasecited by 1
  7. 2026A Context Engineering Framework for Improving Enterprise AI Agents based on Digital-Twin MDPcited by 0
  8. 2026When does a scaling result justify a different allocation? A critical review of resource-allocation evidence for AI systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Likelihood-Based Reward Designs for General LLM Reasoning
  2. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  3. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  4. 2026Enhanced LLM Reasoning by Optimizing Reward Functions with Search-Driven Reinforcement Learning
  5. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  6. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  7. 2025Confidence as a Reward: Transforming LLMs into Reward Models
  8. 2025Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning