mlsanityData to 14 Oct 2026

Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention

arXiv 2609.39494 ↗First posted 30 Sep 2026cs.AI cs.LGcited by 0

On the map: Knowledge distillation → On-policy and self-distillation for LLMs also fits: Knowledge distillation for LLMs

Abstract (arXiv)

Self-distillation with privileged context adapts a language model from demonstrations by letting the model, once conditioned on a reference response, teach its context-free copy token by token. Our taxonomy reveals existing methods differ along three entangled axes: (i) the rollout source (student or teacher), (ii) the teacher coupling (frozen, or an exponential moving average of the student at some coupling rate) and (iii) the KL direction (reverse or forward), yet these axes are usually studied in fixed combinations and have led to conflicting conclusions. We formalize a unifying framework to encompass all self-distillation methods vs classic supervised fine-tuning: we train every combination of the three axes, on Qwen2.5-7B and Ministral-3-3B across ordinary and contradictory tasks, totaling 1,200 adaptation runs, to systematically investigate the impact of the above axes. We propose a controlled model of the same objective to explain the resulting acquisition-retention trade-offs. We find that (i) the rollout source matters mostly where the task contradicts the pretrained behavior: there teacher rollouts raise acquisition well above what student rollouts achieve, with almost no change in retention; (ii) the teacher coupling changes acquisition most, on every task: acquisition rises with the coupling rate, then falls past a task-specific rate; (iii) switching the KL direction costs retention in one model but not the other so which axis to tune first depends on the model. The controlled model reproduces the three trends.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 28 of its 31 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

28 of 31 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents5 Aug
  2. 2026When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation2 Jun
  3. 2026Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation16 May
  4. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents12 Apr
  5. 2026A Survey of On-Policy Distillation for Large Language Models1 Apr
  6. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?25 Mar
  7. 2026Entropy-Aware On-Policy Distillation of Language Models7 Mar
  8. 2026GATES: Self-Distillation under Privileged Context with Consensus Gating24 Feb
  9. 2026Fast and Effective On-policy Distillation from Reasoning Prefixes16 Feb
  10. 2026On-Policy Context Distillation for Language Models12 Feb
  11. 2026Reinforcement Learning via Self-Distillation28 Jan
  12. 2026Self-Distillation Enables Continual Learning27 Jan
1 more
  1. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 15

  1. 2024Qwen2.5 Technical Reportcited by 4,942
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  3. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  4. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  5. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  6. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
  7. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  8. 2022Learning by Distilling Contextcited by 138
  9. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  10. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 94
  11. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 88
  12. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
3 more
  1. 2026Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilitiescited by 3
  2. 2026A Quantitative Characterization of Forgetting in Post-Trainingcited by 3
  3. 2026MixSD: Mixed Contextual Self-Distillation for Knowledge Injectioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Role of Feedback Alignment in Self-Distillation
  2. 2026Distillation Traps and Guards: A Calibration Knob for LLM Distillability
  3. 2026Adaptive Supervised Anchoring for On-Policy Self-Distillation
  4. 2026Self-Distillation with Verbal Regularization
  5. 2022Revisiting Self-Distillation
  6. 2025UNDO: Understanding Distillation as Optimization
  7. 2025Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?
  8. 2026One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context