Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization
On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training
also in 1 more
Abstract (arXiv)
On-Policy Self-distillation is a self-improvement paradigm that enables large language models (LLMs) to improve their capabilities by leveraging the same model as both teacher and student under different conditioning contexts. However, we find that OPSD methods can progressively degrade previously acquired capabilities: queries that are correctly responded at earlier training stages may become incorrectly responded as optimization proceeds, even the overall performance continues to improve. Further analysis reveals that this degradation coincides with a deterioration of the geometry of hidden representations, characterized by reduced intra-cluster compactness and diminished inter-cluster separation. Motivated by this observation, we propose Representation Geometry Regularization (RGR), which preserves representation geometry during post-training by encouraging compactness within cluster and separation across cluster. RGR can be seamlessly integrated into existing OPSD methods without altering their core optimization procedures. Extensive experiments on multiple OPSD methods and benchmarks show that RGR generally alleviates capability degradation while retaining capability improvements. Our code is available in https://github.com/supermachinelearner/RGR.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 18 of its 26 references that are in the map. First posted on arXiv 9 Oct 2026.
Builds on
18 of 26 references.
References · 18
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
- 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
- 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
- 2026Reinforcement Learning via Self-Distillationcited by 332
6 more
- 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
- 2026MiMo-V2-Flash Technical Reportcited by 191
- 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
- 2026Self-Distilled RLVRcited by 131
- 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 99
- 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 76
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting
- 2026GRRR: The Geometry of Reshaping, Rotation, and Routing in Decoder LLM post-training
- 2025LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
- 2026Learning from the Self-future: On-policy Self-distillation for dLLMs
- 2026Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
- 2026OPSDL: On-Policy Self-Distillation for Long-Context Language Models
- 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap
- 2026UniSD: Towards a Unified Self-Distillation Framework for Large Language Models