mlsanityData to 7 Oct 2026

Let's Reinforce Step by Step

arXiv 2311.05821 ↗First posted 10 Nov 2023cs.CLcited by 10

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

While recent advances have boosted LM proficiency in linguistic benchmarks, LMs consistently struggle to reason correctly on complex tasks like mathematics. We turn to Reinforcement Learning from Human Feedback (RLHF) as a method with which to shape model reasoning processes. In particular, we explore two reward schemes, outcome-supervised reward models (ORMs) and process-supervised reward models (PRMs), to optimize for logical reasoning. Our results show that the fine-grained reward provided by PRM-based methods enhances accuracy on simple mathematical reasoning (GSM8K) while, unexpectedly, reducing performance in complex tasks (MATH). Furthermore, we show the critical role reward aggregation functions play in model performance. Providing promising avenues for future research, our study underscores the need for further exploration into fine-grained reward modeling for more reliable language models.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 9 of its 13 references that are in the map. First posted on arXiv 10 Nov 2023.

Builds on

9 of 13 references.

References · 9

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  9. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 5; 2025: 4.

10 papers.

  1. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  2. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  3. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  4. 2025AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verificationcited by 22
  5. 2024From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learningcited by 21
  6. 2025Efficient PRM Training Data Synthesis via Formal Verificationcited by 5
  7. 2025SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modellingcited by 4
  8. 2025ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answeringcited by 3
  9. 2024Adaptive Segment-level Reward: Bridging the Gap Between Action and Reward Space in Alignmentcited by 2
  10. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning
  2. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning
  3. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  4. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
  5. 2025Coarse-to-Fine Process Reward Modeling for Mathematical Reasoning
  6. 2026Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models
  7. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
  8. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning