# Self-Taught Self-Correction for Small Language Models

arXiv: https://arxiv.org/abs/2503.08681 · HTML: https://mlsanity.com/paper/2503.08681/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 13 of 13)

- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity](https://mlsanity.com/paper/2403.14403/) (2024)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023)
- [Generating Sequences by Learning to Self-Correct](https://mlsanity.com/paper/2211.00053/) (2022)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://mlsanity.com/paper/2406.01297/) (2024)
- [REFINER: Reasoning Feedback on Intermediate Representations](https://mlsanity.com/paper/2304.01904/) (2023)
- [Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Home](https://mlsanity.com/paper/2501.12835/) (2025)
- [INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback](https://mlsanity.com/paper/2305.14282/) (2023)
- [Logical Consistency of Large Language Models in Fact-checking](https://mlsanity.com/paper/2412.16100/) (2024)

## Built on by (most cited first, 5 of 5)

- [When Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Models](https://mlsanity.com/paper/2603.23085/) (2026)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [Self-Correcting Code Generation Using Small Language Models](https://mlsanity.com/paper/2505.23060/) (2025)
- [Self-Reflective Generation at Test Time](https://mlsanity.com/paper/2510.02919/) (2025)
- [Beyond Output Critique: Self-Correction via Task Distillation](https://mlsanity.com/paper/2602.00871/) (2026)

## Abstract (arXiv)

Although large language models (LLMs) have achieved remarkable performance across various tasks, they remain prone to errors. A key challenge is enabling them to self-correct. While prior research has relied on external tools or large proprietary models, this work explores self-correction in small language models (SLMs) through iterative fine-tuning using solely self-generated data. We introduce the Self-Taught Self-Correction (STaSC) algorithm, which incorporates multiple algorithmic design choices. Experimental results on a question-answering task demonstrate that STaSC effectively learns self-correction, leading to significant performance improvements. Our analysis further provides insights into the mechanisms of self-correction and the impact of different design choices on learning dynamics and overall performance. To support future research, we release our user-friendly codebase and lightweight models.