mlsanityData to 7 Oct 2026

Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models

arXiv 2412.02674 ↗First posted 3 Dec 2024cs.CL cs.LGcited by 16

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Self-improvement is a mechanism in Large Language Model (LLM) pre-training, post-training and test-time inference. We explore a framework where the model verifies its own outputs, filters or reweights data based on this verification, and distills the filtered data. Despite several empirical successes, a fundamental understanding is still lacking. In this work, we initiate a comprehensive, modular and controlled study on LLM self-improvement. We provide a mathematical formulation for self-improvement, which is largely governed by a quantity which we formalize as the generation-verification gap. Through experiments with various model families and tasks, we discover a scaling phenomenon of self-improvement -- a variant of the generation-verification gap scales monotonically with the model pre-training flops. We also examine when self-improvement is possible, an iterative self-improvement procedure, and ways to improve its performance. Our findings not only advance understanding of LLM self-improvement with practical implications, but also open numerous avenues for future research into its capabilities and boundaries.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 63 of its 94 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

63 of 94 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  2. 2024SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responses4 Apr
  3. 2024Self-Rewarding Language Models18 Jan
  4. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models2 Jan

Other references · 59

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
38 more
  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  4. 2023Teaching Large Language Models to Self-Debugcited by 963
  5. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  7. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  9. 2024Yi: Open Foundation Models by 01.AIcited by 706
  10. 2022Large Language Models Can Self-Improvecited by 656
  11. 2023Textbooks Are All You Needcited by 561
  12. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  13. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  14. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  15. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  16. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  17. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  18. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  19. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  20. 2021Scaling Laws for Transfercited by 265
  21. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  22. 2024Iterative Reasoning Preference Optimizationcited by 244
  23. 2023Self-Consuming Generative Models Go MADcited by 239
  24. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  25. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 156
  26. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
  27. 2024Nemotron-4 340B Technical Reportcited by 143
  28. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  29. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  30. 2024Critique-out-Loud Reward Modelscited by 105
  31. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  32. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
  33. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  34. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  35. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  36. 2024Self-Taught Evaluatorscited by 70
  37. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 67
  38. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  2. 2025Weight Ensembling Improves Reasoning in Language Modelscited by 36
  3. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 19
  4. 2024A Survey on LLM Inference-Time Self-Improvementcited by 17
  5. 2025MetaScale: Test-Time Scaling with Evolving Meta-Thoughtscited by 13
  6. 2025T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Modelscited by 13
  7. 2025Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalizationcited by 12
  8. 2025Aha Moment Revisited: Are VLMs Truly Capable of Self Verification in Inference-time Scaling?cited by 7
  9. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  10. 2026Likelihood-Based Reward Designs for General LLM Reasoningcited by 6
  11. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 6
  12. 2025Can Language Models Falsify? Evaluating Algorithmic Reasoning with Counterexample Creationcited by 5
4 more
  1. 2025Correct Reasoning Paths Visit Shared Decision Pivotscited by 5
  2. 2026On the Generalization Gap in Self-Evolving Language Model Reasoningcited by 4
  3. 2026Peer-Predictive Self-Training for Language Model Reasoningcited by 1
  4. 2026Learning to Draw ASCII Improves Spatial Reasoning in Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap
  2. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  3. 2024Self-Improvement in Language Models: The Sharpening Mechanism
  4. 2025Beyond Accuracy: The Role of Calibration in Self-Improving Large Language Models
  5. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
  6. 2025Self-Improvement in Multimodal Large Language Models: A Survey
  7. 2024Internal Consistency and Self-Feedback in Large Language Models: A Survey
  8. 2025Self-Evolving Critique Abilities in Large Language Models