mlsanityData to 7 Oct 2026

Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations

arXiv 2312.08935 ↗First posted 14 Dec 2023cs.AI cs.CL cs.LGcited by 895

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

In this paper, we present an innovative process-oriented math process reward model called Math-Shepherd, which assigns a reward score to each step of math problem solutions. The training of Math-Shepherd is achieved using automatically constructed process-wise supervision data, breaking the bottleneck of heavy reliance on manual annotation in existing work. We explore the effectiveness of Math-Shepherd in two scenarios: 1) textitVerification: Math-Shepherd is utilized for reranking multiple outputs generated by Large Language Models (LLMs); 2) textitReinforcement Learning: Math-Shepherd is employed to reinforce LLMs with step-by-step Proximal Policy Optimization (PPO). With Math-Shepherd, a series of open-source LLMs demonstrates exceptional performance. For instance, the step-by-step PPO with Math-Shepherd significantly improves the accuracy of Mistral-7B (77.9%→84.1% on GSM8K and 28.6%→33.0% on MATH). The accuracy can be further enhanced to 89.1% and 43.5% on GSM8K and MATH with the verification of Math-Shepherd, respectively. We believe that automatic process supervision holds significant potential for the future evolution of LLMs.

Key facts. Cited by 895 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 49 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

41 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning16 Oct
  2. 2023Let's Verify Step by Step31 May
  3. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 38

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  11. 2023Mistral 7Bcited by 3,065
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
26 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  4. 2023PaLM 2 Technical Reportcited by 1,154
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  7. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  8. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  9. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  10. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  11. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  12. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  13. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  14. 2023Llemma: An Open Language Model For Mathematicscited by 385
  15. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  16. 2023Challenges and Applications of Large Language Modelscited by 251
  17. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  18. 2021Monte Carlo Tree Search: A Review of Recent Modifications and Applicationscited by 158
  19. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  20. 2023Cumulative Reasoning with Large Language Modelscited by 94
  21. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Modelscited by 76
  22. 2023Making Large Language Models Better Reasoners with Alignmentcited by 63
  23. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 50
  24. 2023GRACE: Discriminator-Guided Chain-of-Thought Reasoningcited by 35
  25. 2023When Do Program-of-Thoughts Work for Reasoning?cited by 19
  26. 2023Let's Reinforce Step by Stepcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 98; 2025: 422; 2026: 375.

895 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2025s1: Simple test-time scalingcited by 1,311
  5. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  6. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  7. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  8. 2025Process Reinforcement through Implicit Rewardscited by 478
  9. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  10. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  11. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  12. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
38 more
  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  2. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  3. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  4. 2026Reinforcement Learning via Self-Distillationcited by 285
  5. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  6. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  7. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  8. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  9. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  10. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  11. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  12. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 234
  13. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  14. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  15. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  16. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  17. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
  18. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  19. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
  20. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  21. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuningcited by 167
  22. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  23. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  24. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  25. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  26. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  27. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  28. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  29. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  30. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  31. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  32. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  33. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoningcited by 118
  34. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 117
  35. 2025Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Spacecited by 109
  36. 2026Self-Distilled RLVRcited by 109
  37. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  38. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106

50 most cited of 895 All 895 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Teaching LLMs for Step-Level Automatic Math Correction via Reinforcement Learning
  2. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
  3. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
  4. 2026Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models
  5. 2025VAR-MATH: Probing True Mathematical Reasoning in LLMS via Symbolic Multi-Instance Benchmarks
  6. 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMs
  7. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  8. 2025Right Is Not Enough: The Pitfalls of Outcome Supervision in Training LLMs for Math Reasoning