# S^3cMath: Spontaneous Step-level Self-correction Makes Large Language Models Better Mathematical Reasoners

arXiv: https://arxiv.org/abs/2409.01524 · HTML: https://mlsanity.com/paper/2409.01524/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 16 of 16)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://mlsanity.com/paper/2406.01297/) (2024)
- [LM vs LM: Detecting Factual Errors via Cross Examination](https://mlsanity.com/paper/2305.13281/) (2023)
- [PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations](https://mlsanity.com/paper/2307.02762/) (2023)
- [Common 7B Language Models Already Possess Strong Math Capabilities](https://mlsanity.com/paper/2403.04706/) (2024)
- [Small Language Models Need Strong Verifiers to Self-Correct Reasoning](https://mlsanity.com/paper/2404.17140/) (2024)
- [Augmenting Math Word Problems via Iterative Question Composing](https://mlsanity.com/paper/2401.09003/) (2024)
- [JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models](https://mlsanity.com/paper/2405.14365/) (2024)
- [Evidence-based Factual Error Correction](https://mlsanity.com/paper/2012.15788/) (2020)
- [Can LLMs Solve longer Math Word Problems Better?](https://mlsanity.com/paper/2405.14804/) (2024)
- [DotaMath: Decomposition of Thought with Code Assistance and Self-correction for Mathematical Reasoning](https://mlsanity.com/paper/2407.04078/) (2024)
- [Can We Verify Step by Step for Incorrect Answer Detection?](https://mlsanity.com/paper/2402.10528/) (2024)
- [Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs](https://mlsanity.com/paper/2407.03181/) (2024)

## Built on by (most cited first, 17 of 17)

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges](https://mlsanity.com/paper/2412.11936/) (2024)
- [InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models](https://mlsanity.com/paper/2503.06692/) (2025)
- [Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs](https://mlsanity.com/paper/2412.19513/) (2024)
- [Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization](https://mlsanity.com/paper/2501.17974/) (2025)
- [Let LRMs Break Free from Overthinking via Self-Braking Tuning](https://mlsanity.com/paper/2505.14604/) (2025)
- [LEMMA: Learning from Errors for MatheMatical Advancement in LLMs](https://mlsanity.com/paper/2503.17439/) (2025)
- [Safe: Enhancing Mathematical Reasoning in Large Language Models via Retrospective Step-aware Formal Verification](https://mlsanity.com/paper/2506.04592/) (2025)
- [UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models](https://mlsanity.com/paper/2501.13766/) (2025)
- [AskToAct: Enhancing LLMs Tool Use via Self-Correcting Clarification](https://mlsanity.com/paper/2503.01940/) (2025)
- [RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation](https://mlsanity.com/paper/2502.09183/) (2025)
- [MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task](https://mlsanity.com/paper/2502.11684/) (2025)
- [Teaching LLMs According to Their Aptitude: Adaptive Reasoning for Mathematical Problem Solving](https://mlsanity.com/paper/2502.12022/) (2025)
- [Can Large Reasoning Models Improve Accuracy on Mathematical Tasks Using Flawed Thinking?](https://mlsanity.com/paper/2512.17079/) (2025)
- [From Static to Dynamic: Adaptive Monte Carlo Search for Mathematical Process Supervision](https://mlsanity.com/paper/2509.24351/) (2025)
- [Error Notebook-Guided, Training-Free Part Retrieval in 3D CAD Assemblies via Vision-Language Models](https://mlsanity.com/paper/2509.01350/) (2025)

## Abstract (arXiv)

Self-correction is a novel method that can stimulate the potential reasoning abilities of large language models (LLMs). It involves detecting and correcting errors during the inference process when LLMs solve reasoning problems. However, recent works do not regard self-correction as a spontaneous and intrinsic capability of LLMs. Instead, such correction is achieved through post-hoc generation, external knowledge introduction, multi-model collaboration, and similar techniques. In this paper, we propose a series of mathematical LLMs called S3c-Math, which are able to perform Spontaneous Step-level Self-correction for Mathematical reasoning. This capability helps LLMs to recognize whether their ongoing inference tends to contain errors and simultaneously correct these errors to produce a more reliable response. We proposed a method, which employs a step-level sampling approach to construct step-wise self-correction data for achieving such ability. Additionally, we implement a training strategy that uses above constructed data to equip LLMs with spontaneous step-level self-correction capacities. Our data and methods have been demonstrated to be effective across various foundation LLMs, consistently showing significant progress in evaluations on GSM8K, MATH, and other mathematical benchmarks. To the best of our knowledge, we are the first to introduce the spontaneous step-level self-correction ability of LLMs in mathematical reasoning.