mlsanityData to 14 Oct 2026

Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning

arXiv 2410.08146 ↗First posted 10 Oct 2024cs.LG cs.CLcited by 279

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

A promising approach for improving reasoning in large language models is to use process reward models (PRMs). PRMs provide feedback at each step of a multi-step reasoning trace, potentially improving credit assignment over outcome reward models (ORMs) that only provide feedback at the final step. However, collecting dense, per-step human labels is not scalable, and training PRMs from automatically-labeled data has thus far led to limited gains. To improve a base policy by running search against a PRM or using it as dense rewards for reinforcement learning (RL), we ask: "How should we design process rewards?". Our key insight is that, to be effective, the process reward for a step should measure progress: a change in the likelihood of producing a correct response in the future, before and after taking the step, corresponding to the notion of step-level advantages in RL. Crucially, this progress should be measured under a prover policy distinct from the base policy. We theoretically characterize the set of good provers and our results show that optimizing process rewards from such provers improves exploration during test-time search and online RL. In fact, our characterization shows that weak prover policies can substantially improve a stronger base policy, which we also observe empirically. We validate our claims by training process advantage verifiers (PAVs) to predict progress under such provers, and show that compared to ORMs, test-time search against PAVs is >8% more accurate, and 1.5-5× more compute-efficient. Online RL with dense rewards from PAVs enables one of the first results with 5-6× gain in sample efficiency, and >6% gain in accuracy, over ORMs.

Key facts. Cited by 279 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 48 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

34 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  2. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  3. 2024Teaching Large Language Models to Reason with Reinforcement Learning7 Mar
  4. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning16 Oct
  5. 2023Let's Verify Step by Step31 May
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  8. 2017Thinking Fast and Slow with Deep Learning and Tree Search23 May

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  10. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  11. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  12. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
14 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  2. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  3. 2015Policy Distillationcited by 440
  4. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  5. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  6. 2015MADE: Masked Autoencoder for Distribution Estimationcited by 307
  7. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  8. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  9. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 125
  10. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 118
  11. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  12. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  13. 2015Learning to Search Better Than Your Teachercited by 50
  14. 2023Learning to Generate Better Than Your LLMcited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 159; 2026: 111.

279 papers.

Sort:

By area:

  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  2. 2025Process Reinforcement through Implicit Rewardscited by 504
  3. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  4. 2026Reinforcement Learning via Self-Distillationcited by 332
  5. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  6. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 248
  7. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  8. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 166
  9. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  10. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 136
  11. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  12. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 123
38 more
  1. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 118
  2. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  3. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  4. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  5. 2025How Well do LLMs Compress Their Own Chain-of-Thought? A Token Complexity Approachcited by 97
  6. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 97
  7. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 97
  8. 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 85
  9. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 84
  10. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  11. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 77
  12. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 77
  13. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 76
  14. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  15. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 74
  16. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 73
  17. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 65
  18. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 63
  19. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  20. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 59
  21. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
  22. 2025Evolving Deeper LLM Thinkingcited by 58
  23. 2024The Limits of Inference Scaling Through Resamplingcited by 53
  24. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  25. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  26. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  27. 2025Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendationcited by 51
  28. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 50
  29. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoningcited by 50
  30. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 47
  31. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 46
  32. 2025First Return, Entropy-Eliciting Explorecited by 45
  33. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 42
  34. 2024RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinementcited by 40
  35. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  36. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 40
  37. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 39
  38. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 38

50 most cited of 279 All 279 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
  2. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning
  3. 2026Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models
  4. 2025GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning
  5. 2025R-PRM: Reasoning-Driven Process Reward Modeling
  6. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling
  7. 2026Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning
  8. 2025Process Reward Models That Think