mlsanityData to 7 Oct 2026

Improve Mathematical Reasoning in Language Models by Automated Process Supervision

arXiv 2406.06592 ↗First posted 5 Jun 2024cs.CL cs.LGcited by 426

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Complex multi-step reasoning tasks, such as solving mathematical problems or generating code, remain a significant hurdle for even the most advanced large language models (LLMs). Verifying LLM outputs with an Outcome Reward Model (ORM) is a standard inference-time technique aimed at enhancing the reasoning performance of LLMs. However, this still proves insufficient for reasoning tasks with a lengthy or multi-hop reasoning chain, where the intermediate outcomes are neither properly rewarded nor penalized. Process supervision addresses this limitation by assigning intermediate rewards during the reasoning process. To date, the methods used to collect process supervision data have relied on either human annotation or per-step Monte Carlo estimation, both prohibitively expensive to scale, thus hindering the broad application of this technique. In response to this challenge, we propose a novel divide-and-conquer style Monte Carlo Tree Search (MCTS) algorithm named textitOmegaPRM for the efficient collection of high-quality process supervision data. This algorithm swiftly identifies the first error in the Chain of Thought (CoT) with binary search and balances the positive and negative examples, thereby ensuring both efficiency and quality. As a result, we are able to collect over 1.5 million process supervision annotations to train Process Reward Models (PRMs). This fully automated process supervision alongside the weighted self-consistency algorithm is able to enhance LLMs' math reasoning performances. We improved the success rates of the instruction-tuned Gemini Pro model from 51% to 69.4% on MATH500 and from 86.4% to 93.6% on GSM8K. Similarly, we boosted the success rates of Gemma2 27B from 42.3% to 58.2% on MATH500 and from 74.0% to 92.2% on GSM8K. The entire process operates without any human intervention or supervision, making our method both financially and ...

Key facts. Cited by 426 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 32 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

25 of 32 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision5 Feb
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  3. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning16 Oct
  4. 2023Let's Verify Step by Step31 May
  5. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  11. 2022Large Language Models Can Self-Improvecited by 656
  12. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
8 more
  1. 2021True Few-Shot Learning with Language Modelscited by 343
  2. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  3. 2021Monte Carlo Tree Search: A Review of Recent Modifications and Applicationscited by 158
  4. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  5. 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 84
  6. 2024Augmenting Math Word Problems via Iterative Question Composingcited by 58
  7. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Timecited by 40
  8. 2023Critique Ability of Large Language Modelscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 53; 2025: 248; 2026: 125.

426 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2024Phi-4 Technical Reportcited by 675
  4. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  5. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  6. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  7. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  8. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  9. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  10. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  11. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  12. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
38 more
  1. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  2. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  3. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  4. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  5. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  6. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuningcited by 167
  7. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  8. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  9. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  10. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
  11. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  12. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  13. 2025Process Reward Models That Thinkcited by 118
  14. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
  15. 2026Self-Distilled RLVRcited by 109
  16. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  17. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  18. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  19. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 96
  20. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 94
  21. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 92
  22. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
  23. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  24. 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 81
  25. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  26. 2024Process Reward Model with Q-Value Rankingscited by 78
  27. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 76
  28. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  29. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 75
  30. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  31. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  32. 2025Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learningcited by 68
  33. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  34. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 62
  35. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  36. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 58
  37. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  38. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewardingcited by 53

50 most cited of 426 All 426 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024AlphaMath Almost Zero: Process Supervision without Process
  2. 2024Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning
  3. 2025Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning
  4. 2026Rewarding Novel Deductions: Solver-guided Process Supervision for Logical Reasoning
  5. 2025Coarse-to-Fine Process Reward Modeling for Mathematical Reasoning
  6. 2025Limits of PRM-Guided Tree Search for Mathematical Reasoning with LLMs
  7. 2024Multi-Step Reasoning with Large Language Models, a Survey
  8. 2023No Train Still Gain. Unleash Mathematical Reasoning of Large Language Models with Monte Carlo Tree Search Guided by Energy Function