mlsanityData to 7 Oct 2026

Let's Verify Step by Step

arXiv 2305.20050 ↗First posted 31 May 2023cs.LG cs.AI cs.CLcited by 3,598

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.

Key facts. Cited by 3,598 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 33 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

23 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
10 more
  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  4. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  5. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  6. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  7. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  8. 2021Generate & Rank: A Multi-task Framework for Math Word Problemscited by 82
  9. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  10. 2020Collaborative Storytelling with Large-scale Neural Language Modelscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 112; 2024: 339; 2025: 1307; 2026: 1840.

3,598 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  7. 2025s1: Simple test-time scalingcited by 1,311
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  9. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  11. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  12. 2023Large Language Modelscited by 774
38 more
  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  3. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  4. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  5. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  6. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  7. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  8. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  9. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  10. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  11. 2025Process Reinforcement through Implicit Rewardscited by 478
  12. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  13. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  14. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  15. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  16. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  17. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
  18. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  19. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  20. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  21. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  22. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  23. 2024OmniGen: Unified Image Generationcited by 388
  24. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  25. 2023Llemma: An Open Language Model For Mathematicscited by 385
  26. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  27. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  28. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  29. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  30. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  31. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  32. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  33. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  34. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  35. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
  36. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  37. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  38. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291

50 most cited of 3,598 All 3,598 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Let's Reinforce Step by Step
  2. 2025Efficient Process Reward Model Training via Active Learning
  3. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  4. 2025Right Is Not Enough: The Pitfalls of Outcome Supervision in Training LLMs for Math Reasoning
  5. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
  6. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
  7. 2024PSPO*: An Effective Process-supervised Policy Optimization for Reasoning Alignment
  8. 2024Free Process Rewards without Process Labels