mlsanityData to 14 Oct 2026

A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models

arXiv 2510.08049 ↗First posted 9 Oct 2025cs.CL cs.AIcited by 38

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Although Large Language Models (LLMs) exhibit advanced reasoning ability, conventional alignment remains largely dominated by outcome reward models (ORMs) that judge only final answers. Process Reward Models(PRMs) address this gap by evaluating and guiding reasoning at the step or trajectory level. This survey provides a systematic overview of PRMs through the full loop: how to generate process data, build PRMs, and use PRMs for test-time scaling and reinforcement learning. We summarize applications across math, code, text, multimodal reasoning, robotics, and agents, and review emerging benchmarks. Our goal is to clarify design spaces, reveal open challenges, and guide future research toward fine-grained, robust reasoning alignment.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 83 of its 96 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

83 of 96 references.

Closest ancestors · 41

Same direction, newest first

  1. 2025Rethinking Reward Models for Multi-Domain Test-Time Scaling1 Oct
  2. 2025From Static to Dynamic: Adaptive Monte Carlo Search for Mathematical Process Supervision29 Sep
  3. 2025SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning20 Sep
  4. 2025VRPRM: Process Reward Modeling via Visual Reasoning5 Aug
  5. 2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment4 Aug
  6. 2025The Bidirectional Process Reward Model3 Aug
  7. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner31 Jul
  8. 2025CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoning21 Jul
  9. 2025SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling18 Jun
  10. 2025Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Models11 Jun
  11. 2025FreePRM: Training Process Reward Models Without Ground Truth Process Labels4 Jun
  12. 2025Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns29 May
29 more
  1. 2025Discriminative Policy Optimization for Token-Level Reward Models29 May
  2. 2025Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision26 May
  3. 2025Efficient PRM Training Data Synthesis via Formal Verification21 May
  4. 2025Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs16 May
  5. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning21 Apr
  6. 2025Efficient Process Reward Model Training via Active Learning14 Apr
  7. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning1 Apr
  8. 2025More Bang for the Buck: Process Reward Modeling with Entropy-Driven Uncertainty28 Mar
  9. 2025R-PRM: Reasoning-Driven Process Reward Modeling27 Mar
  10. 2025Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models16 Mar
  11. 2025MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modeling15 Mar
  12. 2025Better Process Supervision with Bi-directional Rewarding Signals6 Mar
  13. 2025Process-based Self-Rewarding Language Models5 Mar
  14. 2025An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning4 Mar
  15. 2025Retrieval-Augmented Process Reward Model for Generalizable Mathematical Reasoning20 Feb
  16. 2025AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence19 Feb
  17. 2025AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verification17 Feb
  18. 2025Process Reward Models for LLM Agents: Practical Framework and Directions14 Feb
  19. 2025VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data10 Feb
  20. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  21. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models6 Jan
  22. 2024Entropy-Regularized Process Reward Model15 Dec
  23. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  24. 2024Process Reward Model with Q-Value Rankings15 Oct
  25. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  26. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  27. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  28. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  29. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 42

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  5. 2024OpenAI o1 System Cardcited by 1,244
  6. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  7. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  8. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  9. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 176
  10. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  11. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 139
  12. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 94
30 more
  1. 2025A Survey of AI Agent Protocolscited by 81
  2. 2025Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learningcited by 70
  3. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 65
  4. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 62
  5. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  6. 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
  7. 2025MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervisioncited by 31
  8. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  9. 2025How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmarkcited by 28
  10. 2025When Agents go Astray: Course-Correcting SWE Agents with PRMscited by 28
  11. 2025Reward-SQL: Boosting Text-to-SQL via Stepwise Execution-Aware Reasoning and Process-Supervised Rewardscited by 27
  12. 2024Process Supervision-Guided Policy Optimization for Code Generationcited by 26
  13. 2025StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Modelscited by 26
  14. 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 26
  15. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 25
  16. 2025MedS3: Towards Medical Slow Thinking with Self-Evolved Soft Dual-sided Process Supervisioncited by 24
  17. 2025ProgRM: Build Better GUI Agents with Progress Rewardscited by 21
  18. 2024Advancing Process Verification for Large Language Models via Tree-Based Preference Learningcited by 16
  19. 2025ViLBench: A Suite for Vision-Language Process Reward Modelingcited by 16
  20. 2025DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoningcited by 15
  21. 2025GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoningcited by 14
  22. 2025MPBench: A Comprehensive Multimodal Reasoning Benchmark for Process Errors Identificationcited by 13
  23. 2025Scaling Evaluation-time Compute with Reasoning Models as Evaluatorscited by 13
  24. 2025PRM-BAS: Enhancing Multimodal Reasoning through PRM-guided Beam Annealing Searchcited by 12
  25. 2025From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignmentcited by 9
  26. 2025Guiding VLM Agents with Process Rewards at Inference Time for GUI Navigationcited by 8
  27. 2025Fin-PRM: A Domain-Specialized Process Reward Model for Financial Reasoning in Large Language Modelscited by 7
  28. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoningcited by 7
  29. 2025Demystifying Multilingual Chain-of-Thought in Process Reward Modelingcited by 6
  30. 2025Test-Time Scaling with Reflective Generative Modelcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 37.

38 papers.

Sort:

By area:

  1. 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineeringcited by 74
  2. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  3. 2026The Art of Building Verifiers for Computer Use Agentscited by 13
  4. 2026Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Modelscited by 8
  5. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  6. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 6
  7. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  8. 2026Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoningcited by 5
  9. 2026Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verificationcited by 5
  10. 2026PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignmentcited by 5
  11. 2026Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysiscited by 5
  12. 2026MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loopcited by 4
26 more
  1. 2026Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Modelscited by 4
  2. 2026Improving Vision-language Models with Perception-centric Process Reward Modelscited by 4
  3. 2026KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scalingcited by 4
  4. 2026More Test-Time Compute Can Hurt: Overestimation Bias in LLM Beam Searchcited by 3
  5. 2026Do We Need Frontier Models to Verify Mathematical Proofs?cited by 3
  6. 2025Reliable agent engineering should integrate machine-compatible organizational principlescited by 2
  7. 2026GeoSolver: Scaling Test-Time Reasoning in Remote Sensing with Fine-Grained Process Supervisioncited by 2
  8. 2026Process Supervision of Confidence Margin for Calibrated LLM Reasoningcited by 2
  9. 2026From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Groundingcited by 2
  10. 2026Training Data Efficiency in Multimodal Process Reward Modelscited by 1
  11. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  12. 2026DiffCold: A Diffusion-based Generative Model for Cold-Start Item Recommendationcited by 1
  13. 2026MetaResearcher: Scaling Deep Research via Self-Reflective Reinforcement Learning in Adversarial Virtual Environmentscited by 1
  14. 2026MAPLE: Elevating Medical Reasoning from Statistical Consensus to Process-Led Alignmentcited by 0
  15. 2026LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Modelscited by 0
  16. 2026GR-Ben: A General Reasoning Benchmark for Evaluating Process Reward Modelscited by 0
  17. 2026DataClawBench: An Agent Benchmark for Exploratory Real-World Financial Data Analysiscited by 0
  18. 2026Learning from Saturated Data: Signals Beyond Correctness for LLM Trainingcited by 0
  19. 2026From Long News to Accurate Forecast: Importance-Aware Fusion and PRM-Guided Reflection for Time Series Forecastingcited by 0
  20. 2026Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMscited by 0
  21. 2026The Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignmentcited by 0
  22. 2026Semantic Drift and the Stability of Operator Control in Reasoning-Class Decision Support Systemscited by 0
  23. 2026NeSyFS: A Neuro-symbolic Fast-Slow Thinking Framework for LLM Agent under Partial Observabilitycited by 0
  24. 2026Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoningcited by 0
  25. 2026Contextual Causality with Large Language Models: A Surveycited by 0
  26. 2026Marathoner: Ultra-Long-Horizon Autonomous Intelligencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning
  2. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling
  3. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  4. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
  5. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs
  6. 2025GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning
  7. 2026Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards
  8. 2024PSPO*: An Effective Process-supervised Policy Optimization for Reasoning Alignment