mlsanityData to 7 Oct 2026

Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision

arXiv 2402.02658 ↗First posted 5 Feb 2024cs.AI cs.CL cs.LGcited by 51

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Process supervision, using a trained verifier to evaluate the intermediate steps generated by a reasoner, has demonstrated significant improvements in multi-step problem solving. In this paper, to avoid the expensive effort of human annotation on the verifier training data, we introduce Model-induced Process Supervision (MiPS), a novel method for automating data curation. MiPS annotates an intermediate step by sampling completions of this solution through the reasoning model, and obtaining an accuracy defined as the proportion of correct completions. Inaccuracies of the reasoner would cause MiPS underestimating the accuracy of intermediate steps, therefore, we suggest and empirically show that verification focusing on high predicted scores of the verifier shall be preferred over that of low predicted scores, contrary to prior observations on human curated data. Our approach significantly improves the performance of PaLM 2 on math and coding tasks (accuracy +0.67% on GSM8K, +4.16% on MATH, +0.92% on MBPP compared with an output supervision trained verifier). Additionally, our study demonstrates that the verifier exhibits strong generalization ability across different reasoning models.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 38 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

30 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  2. 2023Let's Verify Step by Step31 May
  3. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2024Mixtral of Expertscited by 1,785
  12. 2023PaLM 2 Technical Reportcited by 1,154
15 more
  1. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  2. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  3. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  4. 2022Large Language Models Can Self-Improvecited by 656
  5. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  6. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  7. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  8. 2023Llemma: An Open Language Model For Mathematicscited by 385
  9. 2023Language Models can Solve Computer Taskscited by 371
  10. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  11. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  12. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  13. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
  14. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
  15. 2023Critique Ability of Large Language Modelscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 28; 2026: 7.

51 papers.

Sort:

By area:

  1. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  4. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  5. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  6. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  7. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 92
  8. 2025Evolving Deeper LLM Thinkingcited by 58
  9. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  10. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  11. 2024Process-Driven Autoformalization in Lean 4cited by 45
  12. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40
38 more
  1. 2025VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Datacited by 39
  2. 2024Understanding Chain-of-Thought in LLMs through Information Theorycited by 36
  3. 2025ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewardingcited by 32
  4. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  5. 2024AutoPSV: Automated Process-Supervised Verifiercited by 28
  6. 2025MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervisioncited by 28
  7. 2024Process Supervision-Guided Policy Optimization for Code Generationcited by 25
  8. 2025A Tutorial on LLM Reasoning: Relevant Methods behind ChatGPT o1cited by 24
  9. 2024Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlocited by 21
  10. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  11. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  12. 2024FG-PRM: Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoningcited by 12
  13. 2024Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learningcited by 11
  14. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  15. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11
  16. 2025FreePRM: Training Process Reward Models Without Ground Truth Process Labelscited by 10
  17. 2025Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmarkcited by 9
  18. 2025Guided Search Strategies in Non-Serializable Environments with Applications to Software Engineering Agentscited by 9
  19. 2025Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoningcited by 8
  20. 2024Token-Supervised Value Models for Enhancing Mathematical Problem-Solving Capabilities of Large Language Modelscited by 7
  21. 2025Exploring Expert Failures Improves LLM Agent Tuningcited by 7
  22. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  23. 2024Reasoning Paths Optimization: Learning to Reason and Explore From Diverse Pathscited by 6
  24. 2026Verifier-Constrained Flow Expansion for Discovery Beyond the Datacited by 6
  25. 2025Who's Laughing Now? An Overview of Computational Humour Generation and Explanationcited by 5
  26. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  27. 2025SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modellingcited by 4
  28. 2025The Bidirectional Process Reward Modelcited by 4
  29. 2025Lightweight Latent Verifiers for Efficient Meta-Generation Strategiescited by 3
  30. 2025Beyond the First Error: Process Reward Models for Reflective Mathematical Reasoningcited by 3
  31. 2026FunPRM: Function-as-Step Process Reward Model with Meta Reward Correction for Code Generationcited by 3
  32. 2026Self-evolving LLM agents with in-distribution Optimizationcited by 3
  33. 2025AI Agents Should be Regulated Based on the Extent of Their Autonomous Operationscited by 2
  34. 2025RRO: LLM Agent Optimization Through Rising Reward Trajectoriescited by 2
  35. 2026One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignmentcited by 2
  36. 2025DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Trainingcited by 1
  37. 2025Enhancing LLM Reasoning via Non-Human-Like Reasoning Path Preference Optimizationcited by 0
  38. 2026MAPLE: Elevating Medical Reasoning from Statistical Consensus to Process-Led Alignmentcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Guiding Through Complexity: What Makes Good Supervision for Hard Math Reasoning Tasks?
  2. 2025An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning
  3. 2025Dyve: Thinking Fast and Slow for Dynamic Process Verification
  4. 2024V-STaR: Training Verifiers for Self-Taught Reasoners
  5. 2025EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoning
  6. 2024RATIONALYST: Mining Implicit Rationales for Process Supervision of Reasoning
  7. 2025Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search
  8. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning