# Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models

arXiv: https://arxiv.org/abs/2412.02674 · HTML: https://mlsanity.com/paper/2412.02674/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 63 of its 94 references that are in the map.
- First posted on arXiv 3 Dec 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-rewarding alignment and self-training](https://mlsanity.com/synth/self-rewarding-alignment-and-self-training/)

## Builds on (most cited first, 30 of 63)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [SLiC-HF: Sequence Likelihood Calibration with Human Feedback](https://mlsanity.com/paper/2305.10425/) (2023)

## Built on by (most cited first, 18 of 18)

- [Scaling Test-Time Compute Without Verification or RL is Suboptimal](https://mlsanity.com/paper/2502.12118/) (2025)
- [Weight Ensembling Improves Reasoning in Language Models](https://mlsanity.com/paper/2504.10478/) (2025)
- [Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning](https://mlsanity.com/paper/2511.19942/) (2025)
- [A Survey on LLM Inference-Time Self-Improvement](https://mlsanity.com/paper/2412.14352/) (2024)
- [Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalization](https://mlsanity.com/paper/2511.07378/) (2025)
- [MetaScale: Test-Time Scaling with Evolving Meta-Thoughts](https://mlsanity.com/paper/2503.13447/) (2025)
- [T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Models](https://mlsanity.com/paper/2504.04718/) (2025)
- [SPADE: Self-Play in Adaptive Synthetic Executable Environments](https://mlsanity.com/paper/2608.19197/) (2026)
- [Aha Moment Revisited: Are VLMs Truly Capable of Self Verification in Inference-time Scaling?](https://mlsanity.com/paper/2506.17417/) (2025)
- [Can Language Models Falsify? Evaluating Algorithmic Reasoning with Counterexample Creation](https://mlsanity.com/paper/2502.19414/) (2025)
- [LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learning](https://mlsanity.com/paper/2602.01705/) (2026)
- [Likelihood-Based Reward Designs for General LLM Reasoning](https://mlsanity.com/paper/2602.03979/) (2026)
- [Correct Reasoning Paths Visit Shared Decision Pivots](https://mlsanity.com/paper/2509.21549/) (2025)
- [On the Generalization Gap in Self-Evolving Language Model Reasoning](https://mlsanity.com/paper/2606.01075/) (2026)
- [Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models](https://mlsanity.com/paper/2607.12962/) (2026)
- [Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets](https://mlsanity.com/paper/2608.01000/) (2026)
- [Peer-Predictive Self-Training for Language Model Reasoning](https://mlsanity.com/paper/2604.13356/) (2026)
- [Learning to Draw ASCII Improves Spatial Reasoning in Language Models](https://mlsanity.com/paper/2604.14641/) (2026)

## Abstract (arXiv)

Self-improvement is a mechanism in Large Language Model (LLM) pre-training, post-training and test-time inference. We explore a framework where the model verifies its own outputs, filters or reweights data based on this verification, and distills the filtered data. Despite several empirical successes, a fundamental understanding is still lacking. In this work, we initiate a comprehensive, modular and controlled study on LLM self-improvement. We provide a mathematical formulation for self-improvement, which is largely governed by a quantity which we formalize as the generation-verification gap. Through experiments with various model families and tasks, we discover a scaling phenomenon of self-improvement -- a variant of the generation-verification gap scales monotonically with the model pre-training flops. We also examine when self-improvement is possible, an iterative self-improvement procedure, and ways to improve its performance. Our findings not only advance understanding of LLM self-improvement with practical implications, but also open numerous avenues for future research into its capabilities and boundaries.