mlsanityData to 7 Oct 2026

Self-Distillation Enables Continual Learning

arXiv 2601.19897 ↗First posted 27 Jan 2026cs.LGcited by 212

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models. While on-policy reinforcement learning can reduce forgetting, it requires explicit reward functions that are often unavailable. Learning from expert demonstrations, the primary alternative, is dominated by supervised fine-tuning (SFT), which is inherently off-policy. We introduce Self-Distillation Fine-Tuning (SDFT), a simple method that enables on-policy learning directly from demonstrations. SDFT leverages in-context learning by using a demonstration-conditioned model as its own teacher, generating on-policy training signals that preserve prior capabilities while acquiring new skills. Across skill learning and knowledge acquisition tasks, SDFT consistently outperforms SFT, achieving higher new-task accuracy while substantially reducing catastrophic forgetting. In sequential learning experiments, SDFT enables a single model to accumulate multiple skills over time without performance regression, establishing on-policy distillation as a practical path to continual learning from demonstrations.

Key facts. Cited by 212 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 60 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

46 of 60 references.

References · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2015Trust Region Policy Optimizationcited by 2,503
34 more
  1. 2016Learning without Forgettingcited by 2,465
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  7. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  8. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  9. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  10. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  11. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  12. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  13. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  14. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  15. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  16. 2022Learning by Distilling Contextcited by 128
  17. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  18. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  19. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
  20. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  21. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 90
  22. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  23. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  24. 2024Injecting New Knowledge into Large Language Models via Supervised Fine-Tuningcited by 56
  25. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 42
  26. 2022Sequence Model Imitation Learning with Unobserved Contextscited by 34
  27. 2023Propagating Knowledge Updates to LMs Through Distillationcited by 28
  28. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 27
  29. 2024Efficient Knowledge Injection in LLMs via Self-Distillationcited by 22
  30. 2025InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformationcited by 16
  31. 2025Unveiling the Compositional Ability Gap in Vision-Language Reasoning Modelcited by 13
  32. 2025Better Estimation of the Kullback--Leibler Divergence Between Language Modelscited by 10
  33. 2025Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learningcited by 9
  34. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approachcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 212.

212 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 285
  2. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  3. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  4. 2026Self-Distilled RLVRcited by 109
  5. 2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 105
  6. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 87
  7. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  8. 2026Privileged Information Distillation for Language Modelscited by 79
  9. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 65
  10. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 58
  11. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  12. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 49
38 more
  1. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  2. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 36
  3. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 29
  4. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  5. 2026Rethinking On-Policy Self-Distillation for Thinking Modelscited by 28
  6. 2026PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competencecited by 26
  7. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 24
  8. 2026Aligning Language Models from User Interactionscited by 21
  9. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Whycited by 21
  10. 2026D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Modelscited by 17
  11. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
  12. 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domainscited by 17
  13. 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 15
  14. 2026Trajectory-Refined Distillationcited by 14
  15. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 13
  16. 2026OPSD-V: On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generatorscited by 12
  17. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillationcited by 12
  18. 2026Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillationcited by 11
  19. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  20. 2026Reward-Gated On-Policy Distillationcited by 11
  21. 2026On-Policy Self-Distillation without Any Supervisioncited by 11
  22. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 10
  23. 2026V-Zero: Answer-Label-Free On-Policy Distillation with Contrastive Evidence Gating for Fine-Grained Visual Reasoningcited by 10
  24. 2026UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillationcited by 10
  25. 2026Pass the Baton: Trajectory-Relayed On-Policy Distillationcited by 10
  26. 2026Learning to Hint for Reinforcement Learningcited by 9
  27. 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 9
  28. 2026VISD: Enhancing Video Reasoning via Structured Self-Distillationcited by 9
  29. 2026Adaptive Teacher Exposure for Self-Distillation in LLM Reasoningcited by 9
  30. 2026From Generic Correlation to Input-Specific Credit in On-Policy Self Distillationcited by 9
  31. 2026Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Trainingcited by 9
  32. 2026Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoningcited by 9
  33. 2026OPRD: On-Policy Representation Distillationcited by 9
  34. 2026DOPD: Dual On-policy Distillationcited by 9
  35. 2026Visual Contrastive Self-Distillationcited by 9
  36. 2026Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Trainingcited by 8
  37. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  38. 2026Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillationcited by 8

50 most cited of 212 All 212 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning without Overwriting: A Theory of Self-Distillation and Supervised Fine-Tuning in Continual Reasoning
  2. 2026Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning
  3. 2025Beyond Imitation: Recovering Dense Rewards from Demonstrations
  4. 2026Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning
  5. 2026Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation
  6. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents
  7. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners
  8. 2022Strategy Discovery and Mixture in Lifelong Learning from Heterogeneous Demonstration