mlsanityData to 14 Oct 2026

Solving math word problems with process- and outcome-based feedback

arXiv 2211.14275 ↗First posted 25 Nov 2022cs.LG cs.AI cs.CLcited by 901

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent work has shown that asking language models to generate reasoning steps improves performance on many reasoning tasks. When moving beyond prompting, this raises the question of how we should supervise such models: outcome-based approaches which supervise the final result, or process-based approaches which supervise the reasoning process itself? Differences between these approaches might naturally be expected not just in final-answer errors but also in reasoning errors, which can be difficult to detect and are problematic in many real-world domains such as education. We run the first comprehensive comparison between process- and outcome-based approaches trained on a natural language task, GSM8K. We find that pure outcome-based supervision produces similar final-answer error rates with less label supervision. However, for correct reasoning steps we find it necessary to use process-based supervision or supervision from learned reward models that emulate process-based feedback. In total, we improve the previous best results from 16.8% → 12.7% final-answer error and 14.0% → 3.4% reasoning error among final-answer-correct solutions.

Key facts. Cited by 901 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 65 references that are in the map. First posted on arXiv 25 Nov 2022.

Builds on

52 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Thinking Fast and Slow with Deep Learning and Tree Search23 May

Other references · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
38 more
  1. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  5. 2016Concrete Problems in AI Safetycited by 1,642
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  10. 2017Selective Classification for Deep Neural Networkscited by 852
  11. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  12. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  13. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  14. 2014Neural Turing Machinescited by 644
  15. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  16. 2018AI safety via debatecited by 400
  17. 2020Generative Language Modeling for Automated Theorem Provingcited by 388
  18. 2021True Few-Shot Learning with Language Modelscited by 344
  19. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  20. 2018Maximum a Posteriori Policy Optimisationcited by 328
  21. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
  22. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  23. 2022Teaching language models to support answers with verified quotescited by 292
  24. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  25. 2021Alignment of Language Agentscited by 168
  26. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  27. 2021Explaining Answers with Entailment Treescited by 152
  28. 2018Supervising strong learners by amplifying weak expertscited by 145
  29. 2020Unsupervised Question Decomposition for Question Answeringcited by 118
  30. 2022Language Model Cascadescited by 107
  31. 2015Neural Programmer-Interpreterscited by 101
  32. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 70
  33. 2019An investigation of model-free planningcited by 66
  34. 2022Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Modelscited by 49
  35. 2020Measuring Systematic Generalization in Neural Proof Generation with Transformerscited by 43
  36. 2017Making Neural Programming Architectures Generalize via Recursioncited by 35
  37. 2017Weakly-supervised Semantic Parsing with Abstract Examplescited by 13
  38. 2020Avoiding Tampering Incentives in Deep RL via Decoupled Approvalcited by 12

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 55; 2024: 125; 2025: 325; 2026: 395.

901 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Let's Verify Step by Stepcited by 3,789
  3. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  4. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  6. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  9. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  10. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  11. 2025Process Reinforcement through Implicit Rewardscited by 504
  12. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
38 more
  1. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  2. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  3. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  4. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  5. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  6. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  7. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  8. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  9. 2023Llemma: An Open Language Model For Mathematicscited by 388
  10. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  11. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  12. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  13. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  14. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  15. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  16. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  17. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  18. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  19. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  20. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 279
  21. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 249
  22. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  23. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  24. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  25. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 239
  26. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  27. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  28. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  29. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  30. 2024Chain-of-Thought Reasoning Without Promptingcited by 210
  31. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 196
  32. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 187
  33. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  34. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  35. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  36. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
  37. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
  38. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160

50 most cited of 901 All 901 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Right Is Not Enough: The Pitfalls of Outcome Supervision in Training LLMs for Math Reasoning
  2. 2023Let's Reinforce Step by Step
  3. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning
  4. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
  5. 2026Rewarding Novel Deductions: Solver-guided Process Supervision for Logical Reasoning
  6. 2026InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning
  7. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process
  8. 2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems