mlsanityData to 7 Oct 2026

Enhancing Mathematical Reasoning in LLMs by Stepwise Correction

arXiv 2410.12934 ↗First posted 16 Oct 2024cs.CLcited by 15

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Best-of-N decoding methods instruct large language models (LLMs) to generate multiple solutions, score each using a scoring function, and select the highest scored as the final answer to mathematical reasoning problems. However, this repeated independent process often leads to the same mistakes, making the selected solution still incorrect. We propose a novel prompting method named Stepwise Correction (StepCo) that helps LLMs identify and revise incorrect steps in their generated reasoning paths. It iterates verification and revision phases that employ a process-supervised verifier. The verify-then-revise process not only improves answer correctness but also reduces token consumption with fewer paths needed to generate. With StepCo, a series of LLMs demonstrate exceptional performance. Notably, using GPT-4o as the backend LLM, StepCo achieves an average accuracy of 94.1 across eight datasets, significantly outperforming the state-of-the-art Best-of-N method by +2.4, while reducing token consumption by 77.8%.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 42 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

30 of 42 references.

References · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  11. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  12. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
18 more
  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  3. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  4. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  5. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  6. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  7. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  8. 2023Language Models can Solve Computer Taskscited by 371
  9. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  10. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  11. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  12. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  13. 2021Monte Carlo Tree Search: A Review of Recent Modifications and Applicationscited by 158
  14. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 157
  15. 2018Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problemscited by 49
  16. 2020Graph-to-Tree Neural Networks for Learning Structured Input-Output Translation with Applications to Semantic Parsing and Math Word Problemcited by 31
  17. 2020Learning by Fixing: Solving Math Word Problems with Weak Supervisioncited by 25
  18. 2020SMART: A Situation Model for Algebra Story Problems via Attributed Grammarcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 4.

15 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  3. 2025ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewardingcited by 32
  4. 2025Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoningcited by 14
  5. 2025Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignmentcited by 12
  6. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  7. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  8. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  9. 2026DiffCoT: Diffusion-styled Chain-of-Thought Reasoning in LLMscited by 5
  10. 2025Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategiescited by 2
  11. 2025Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuningcited by 2
  12. 2025TextualVerifier: Verify TextGrad Step-by-Stepcited by 2
3 more
  1. 2026SafeMath: Safe Solutions for Unsafe Math Word Problemscited by 0
  2. 2026Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decodingcited by 0
  3. 2026When Tokenization is Secretly Output Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedback
  2. 2024Step Guided Reasoning: Improving Mathematical Reasoning using Guidance Generation and Step Reasoning
  3. 2024S^3cMath: Spontaneous Step-level Self-correction Makes Large Language Models Better Mathematical Reasoners
  4. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification
  5. 2022Large Language Models are Better Reasoners with Self-Verification
  6. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
  7. 2026Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution
  8. 2024Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoning