mlsanityData to 7 Oct 2026

Let's reward step by step: Step-Level reward model as the Navigators for Reasoning

arXiv 2310.10080 ↗First posted 16 Oct 2023cs.CLcited by 96

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent years have seen considerable advancements in multi-step reasoning with Large Language Models (LLMs). The previous studies have elucidated the merits of integrating feedback or search mechanisms during model inference to improve the reasoning accuracy. The Process-Supervised Reward Model (PRM), typically furnishes LLMs with step-by-step feedback during the training phase, akin to Proximal Policy Optimization (PPO) or reject sampling. Our objective is to examine the efficacy of PRM in the inference phase to help discern the optimal solution paths for multi-step tasks such as mathematical reasoning and code generation. To this end, we propose a heuristic greedy search algorithm that employs the step-level feedback from PRM to optimize the reasoning pathways explored by LLMs. This tailored PRM demonstrated enhanced results compared to the Chain of Thought (CoT) on mathematical benchmarks like GSM8K and MATH. Additionally, to explore the versatility of our approach, we develop a novel method to automatically generate step-level reward dataset for coding tasks and observed similar improved performance in the code generation tasks. Thus highlighting the robust nature of our reward-model-based approach to inference for reasoning tasks.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 54 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

42 of 54 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  2. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  3. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
26 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2023Teaching Large Language Models to Self-Debugcited by 963
  8. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  9. 2023StarCoder: may the source be with you!cited by 790
  10. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  11. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  12. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  13. 2022PAL: Program-aided Language Modelscited by 635
  14. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  15. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  16. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  17. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  18. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  19. 2016Solving General Arithmetic Word Problemscited by 353
  20. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  21. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  22. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 195
  23. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  24. 2023Self-Verification Improves Few-Shot Clinical Information Extractioncited by 52
  25. 2023Automatic Model Selection with Large Language Models for Reasoningcited by 35
  26. 2023LeTI: Learning to Generate from Textual Interactionscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 23; 2025: 43; 2026: 28.

96 papers.

Sort:

By area:

  1. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  2. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  4. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  5. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  6. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  7. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  8. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  9. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
  10. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
  11. 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 81
  12. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 77
38 more
  1. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  2. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 74
  3. 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
  4. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  5. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoningcited by 58
  6. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  7. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  8. 2024Process-Driven Autoformalization in Lean 4cited by 45
  9. 2024Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Treescited by 38
  10. 2024Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoningcited by 38
  11. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  12. 2024LiteSearch: Efficacious Tree Search for LLMcited by 30
  13. 2025Benchmarking Reasoning Robustness in Large Language Modelscited by 30
  14. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 29
  15. 2024AutoPSV: Automated Process-Supervised Verifiercited by 28
  16. 2025MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervisioncited by 28
  17. 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 25
  18. 2023Mathematical Language Models: A Surveycited by 24
  19. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratchcited by 24
  20. 2025AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidencecited by 23
  21. 2024StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learningcited by 22
  22. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  23. 2025Process-Supervised Reinforcement Learning for Code Generationcited by 21
  24. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  25. 2025Multi-Step Visual Reasoning with Visual Tokens Scaling and Verificationcited by 19
  26. 2025Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scalingcited by 15
  27. 2024What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoningcited by 14
  28. 2025Conformal Language Model Reasoning with Coherent Factualitycited by 14
  29. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  30. 2025Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMscited by 12
  31. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  32. 2025Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agentscited by 11
  33. 2025On the Query Complexity of Verifier-Assisted Language Generationcited by 10
  34. 2025Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learningcited by 10
  35. 2025Thinking Preference Optimizationcited by 8
  36. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 8
  37. 2025Can Test-Time Scaling Improve World Foundation Model?cited by 7
  38. 2025Exploring Expert Failures Improves LLM Agent Tuningcited by 7

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling
  2. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling
  3. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
  4. 2025R-PRM: Reasoning-Driven Process Reward Modeling
  5. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning
  6. 2026Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards
  7. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning
  8. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps