mlsanityData to 14 Oct 2026

Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap

arXiv 2507.00075 ↗First posted 29 Jun 2025cs.LG cs.AIcited by 6

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Self-improvement is a significant techniques within the realm of large language model (LLM), aiming to enhance the LLM performance without relying on external data. Despite its significance, generally how LLM performances evolve during the self-improvement process remains underexplored. In this paper, we theoretically model the training dynamics of self-improvement via the concept of solver-verifier gap. This is inspired by the conjecture that the performance enhancement of self-improvement stems from the gap between LLM's solver capability and verifier capability. Based on the theoretical framework, we further show how to model the entire training trajectory. This framework allows quantifying the capability limit of self-improvement by fitting the theoretical model to the experiment results. We validate the effectiveness of the theoretical framework on various LLMs and datasets. Beyond self-improvement, we extend our analysis to investigate how external data influences these dynamics within the framework. Notably, we find that under limited external data regimes, such external data can be utilized at any stage without significantly affecting final performances, which accords with the empirical observations.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 53 of its 60 references that are in the map. First posted on arXiv 29 Jun 2025.

Builds on

53 of 60 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Superficial Self-Improved Reasoners Benefit from Model Merging3 Mar
  2. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  3. 2024Self-Improvement in Language Models: The Sharpening Mechanism2 Dec
  4. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models21 Jun
  5. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing18 Apr
  6. 2023Language Model Self-improvement by Reinforcement Learning Contemplation23 May

Other references · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2023Let's Verify Step by Stepcited by 3,789
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
35 more
  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  4. 2022Large Language Models Can Self-Improvecited by 664
  5. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  6. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  7. 2023Scaling Data-Constrained Language Modelscited by 403
  8. 2022Large Language Models Are Reasoning Teacherscited by 329
  9. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  10. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  11. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
  12. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 151
  13. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  14. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 100
  15. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
  16. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  17. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 70
  18. 2024Variational Best-of-N Alignmentcited by 51
  19. 2023Learning to Generate Better Than Your LLMcited by 49
  20. 2024Large Language Models have Intrinsic Self-Correction Abilitycited by 46
  21. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 44
  22. 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 43
  23. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 41
  24. 2024Towards Theoretical Understandings of Self-Consuming Generative Modelscited by 29
  25. 2023Understanding Self-Distillation in the Presence of Label Noisecited by 21
  26. 2021Self-training Converts Weak Learners to Strong Learners in Mixture Modelscited by 19
  27. 2024Self-Improving Robust Preference Optimizationcited by 19
  28. 2024ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvementcited by 18
  29. 2025EvoLM: In Search of Lost Language Model Training Dynamicscited by 16
  30. 2024Towards a theory of model distillationcited by 15
  31. 2024Understanding the Gains from Repeated Self-Distillationcited by 15
  32. 2022Eliminating Meta Optimization Through Self-Referential Meta Learningcited by 14
  33. 2015From Seed AI to Technological Singularity via Recursively Self-Improving Softwarecited by 9
  34. 2025Will LLMs Scaling Hit the Wall? Breaking Barriers via Distributed Resources on Massive Edge Devicescited by 9
  35. 2025Augmenting Human-Annotated Training Data with Large Language Model Generation and Distillation in Open-Response Assessmentcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalizationcited by 14
  2. 2026On the Generalization Gap in Self-Evolving Language Model Reasoningcited by 4
  3. 2026Peer-Predictive Self-Training for Language Model Reasoningcited by 1
  4. 2026When Good Verifiers Go Bad: Silent Negative Transfer in Verifier-Guided VLM Trainingcited by 0
  5. 2026When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gaincited by 0
  6. 2026Provable Test-Time Scaling for Beam Search in LLM Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
  2. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  3. 2025Beyond Accuracy: The Role of Calibration in Self-Improving Large Language Models
  4. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
  5. 2026Self-Improving Large Language Models via Progressive Experience Evolution
  6. 2024Can Large Language Models Invent Algorithms to Improve Themselves?: Algorithm Discovery for Recursive Self-Improvement through Reinforcement Learning
  7. 2025Self-Improvement in Multimodal Large Language Models: A Survey
  8. 2025Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimization