mlsanityData to 7 Oct 2026

Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training

arXiv 2606.21090 ↗First posted 17 Jun 2026cs.AI cs.LGcited by 4

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Self-improvement can self-regress. In REINFORCE post-training for code, a model can quickly improve on its optimized metric and then collapse within the same training campaign. We study this in a controlled multi-seed testbed using Qwen-2.5-3B and Qwen-2.5-7B, trained on competitive-programming tasks with binary CodeGrader reward across 10 sequential 20-step campaigns. Across campaigns, pass@1 shows a robust rise-then-collapse pattern: it peaks within tens of gradient steps and then falls back, sometimes to near zero. This is not cross-task catastrophic forgetting, but within-task policy over-optimization on a fixed distribution; KL- and EWC-style constraints do not prevent it. We ask where the control loop should sit. We compare three levels: CARE, a between-campaign memory mechanism with a capability posterior, transfer gate, and regression-aware belief revision; ES, a within-campaign early-stop rule that rolls forward the peak checkpoint and sets the next budget to peak_step+3; and GRPO, which changes the RL update using group-relative reward normalization. The answer is regime-dependent. On Qwen-2.5-3B, where naive REINFORCE is fragile, CARE v2 nearly doubles end-of-chain pass@1 from 4.9% to 9.5%, with paired bootstrap 95% CI [+0.4,+8.9] and gains in 4/5 seeds. On Qwen-2.5-7B, CARE reaches parity with naive REINFORCE, 13.8% vs. 11.8%, while ES reaches 22.2% [14.1,28.0]. Out-of-the-box GRPO reaches 20.7% [15.7,25.1], nearly matching REINFORCE+ES. GRPO raises the floor but does not remove the cliff. Its 7B gain mainly comes from better between-campaign carryover, while the within-campaign peak-to-end gap remains about 17 points under both REINFORCE and GRPO. GRPO+ES gives mixed evidence: 2/3 seeds improve, but one final cliff lowers the mean to 17.0% [0.0,28.1]. A Gemma-3-4B pilot shows the same signature, suggesting the phenomenon is not limited to Qwen.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 39 references that are in the map. First posted on arXiv 17 Jun 2026.

Builds on

29 of 39 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective10 Feb
  2. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward18 Dec
  3. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?1 Oct
  4. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  5. 2025Can Large Reasoning Models Self-Train?27 May
  6. 2024Progress or Regress? Self-Improvement Reversal in Post-training6 Jul
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  8. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  10. 2021Measuring Coding Challenge Competence With APPScited by 879
  11. 2024Self-Rewarding Language Modelscited by 654
  12. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
9 more
  1. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  2. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  3. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 377
  4. 2017Population Based Training of Neural Networkscited by 351
  5. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  6. 2024Iterative Reasoning Preference Optimizationcited by 244
  7. 2020Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimizationcited by 176
  8. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  9. 2022Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overviewcited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 25
  2. 2026Training AI Scientists to Replicate Researchcited by 4
  3. 2026Question Begets Question: Self-Evolving Curriculum for Reinforcement Fine-Tuning on Competition Mathematicscited by 2
  4. 2026When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Bootstrapping Task Spaces for Self-Improvement
  2. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap
  3. 2026R2 Flow: Recursive Self-Improvement via Recursive Skill Evolution
  4. 2026Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages
  5. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
  6. 2026A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula
  7. 2026SAHOO: Safeguarded Alignment for High-Order Optimization Objectives in Recursive Self-Improvement
  8. 2026Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation