mlsanityData to 7 Oct 2026

The Lessons of Developing Process Reward Models in Mathematical Reasoning

arXiv 2501.07301 ↗First posted 13 Jan 2025cs.CL cs.AI cs.LGcited by 402

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Process Reward Models (PRMs) emerge as a promising approach for process supervision in mathematical reasoning of Large Language Models (LLMs), which aim to identify and mitigate intermediate errors in the reasoning processes. However, the development of effective PRMs faces significant challenges, particularly in data annotation and evaluation methodologies. In this paper, through extensive experiments, we demonstrate that commonly used Monte Carlo (MC) estimation-based data synthesis for PRMs typically yields inferior performance and generalization compared to LLM-as-a-judge and human annotation methods. MC estimation relies on completion models to evaluate current-step correctness, leading to inaccurate step verification. Furthermore, we identify potential biases in conventional Best-of-N (BoN) evaluation strategies for PRMs: (1) The unreliable policy models generate responses with correct answers but flawed processes, leading to a misalignment between the evaluation criteria of BoN and the PRM objectives of process verification. (2) The tolerance of PRMs of such responses leads to inflated BoN scores. (3) Existing PRMs have a significant proportion of minimum scores concentrated on the final answer steps, revealing the shift from process to outcome-based assessment in BoN Optimized PRMs. To address these challenges, we develop a consensus filtering mechanism that effectively integrates MC estimation with LLM-as-a-judge and advocates a more comprehensive evaluation framework that combines response-level and step-level metrics. Based on the mechanisms, we significantly improve both model performance and data efficiency in the BoN evaluation and the step-wise error identification task. Finally, we release a new state-of-the-art PRM that outperforms existing open-source alternatives and provides practical guidelines for future research in building process supervision models.

Key facts. Cited by 402 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 36 references that are in the map. First posted on arXiv 13 Jan 2025.

Builds on

30 of 36 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  2. 2024Free Process Rewards without Process Labels2 Dec
  3. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  4. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  5. 2024AlphaMath Almost Zero: Process Supervision without Process6 May
  6. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning8 Feb
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  8. 2023Let's Verify Step by Step31 May
  9. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2024GPT-4o System Cardcited by 4,467
  10. 2024Qwen2 Technical Reportcited by 2,076
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
9 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  2. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  3. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  4. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  5. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 95
  6. 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 89
  7. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  8. 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
  9. 2024LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedbackcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 238; 2026: 164.

402 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  4. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  5. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  6. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
  7. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  8. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuningcited by 167
  9. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithfulcited by 166
  10. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  11. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  12. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
38 more
  1. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
  2. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  3. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  4. 2025Qwen3Guard Technical Reportcited by 125
  5. 2025Process Reward Models That Thinkcited by 118
  6. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  7. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  8. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
  9. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 94
  10. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 93
  11. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 89
  12. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 77
  13. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 74
  14. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  15. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
  16. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  17. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoningcited by 49
  18. 2025Reward Reasoning Modelcited by 47
  19. 2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Rewardcited by 45
  20. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
  21. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 44
  22. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
  23. 2025MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought Verificationcited by 39
  24. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 39
  25. 2025R-PRM: Reasoning-Driven Process Reward Modelingcited by 38
  26. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 37
  27. 2025Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!cited by 36
  28. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  29. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  30. 2026MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wildcited by 35
  31. 2026DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Modelscited by 35
  32. 2025Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoningcited by 33
  33. 2025Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoningcited by 32
  34. 2025Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspectivecited by 32
  35. 2025Learning to Reason Across Parallel Samples for LLM Reasoningcited by 32
  36. 2025ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Taskscited by 30
  37. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  38. 2025MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervisioncited by 28

50 most cited of 402 All 402 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning
  2. 2025An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning
  3. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models
  4. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  5. 2023Let's Reinforce Step by Step
  6. 2026Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards
  7. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
  8. 2025Coarse-to-Fine Process Reward Modeling for Mathematical Reasoning