mlsanityData to 14 Oct 2026

Continually Post-training Small Models via Synthetic Self-distillation

arXiv 2610.13113 ↗First posted 9 Oct 2026cs.LG cs.AIcited by 0

On the map: Knowledge distillation → Distillation for continual learning also fits: On-policy and self-distillation for LLMs

Abstract (arXiv)

Continual post-training faces a dual challenge of acquisition and retention: models must learn new capabilities while preserving prior knowledge. While Self-Distillation Fine-Tuning (SDFT) mitigates this using self-supervision with in-context demonstrations, it relies on strong in-context learning and instruction-following abilities. Consequently, SDFT fails on smaller models critical for low-latency, on-device use. To enable continual learning at smaller scales, we first identify the root cause of failure. We show that poor acquisition and forgetting correlate linearly with the post-trained model's distribution shift, revealing the necessity of in-distribution learning. We thus introduce Synthetic Self-Distillation (SynthSelf), imitating SDFT's in-distribution rewrite by outsourcing the task to a minimally larger, same-family aligner. Coupled with off-policy supervision and rejection sampling, this provides the high-quality supervision small models require without breaking their original distribution. Evaluations spanning 4 acquisition datasets (including a mixture) and 14 retention benchmarks show SynthSelf consistently outperforms 6 methods (including off- and on-policy distillation) across 3 small model scales. Extensive further analyses confirm SynthSelf paves the way toward effective small-model continual learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Distillation for continual learning, one of 12 directions in Knowledge distillation. Builds on 32 of its 40 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

32 of 40 references.

References · 32

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  10. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  11. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  12. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
20 more
  1. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  2. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  3. 2025Small Language Models are the Future of Agentic AIcited by 270
  4. 2026Self-Distillation Enables Continual Learningcited by 247
  5. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  6. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  7. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  8. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  9. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 94
  10. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 64
  11. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 58
  12. 2025Spurious Forgetting in Continual Learning of Language Modelscited by 51
  13. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 47
  14. 2024Scaling Laws for Forgetting When Fine-Tuning Large Language Modelscited by 38
  15. 2025Anchored Supervised Fine-Tuningcited by 15
  16. 2025Mapping Post-Training Forgetting in Language Models at Scalecited by 11
  17. 2026How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Datacited by 8
  18. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 6
  19. 2026RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgettingcited by 3
  20. 2026Think Big, Search Small: Where Capacity Matters in Hierarchical Search Agents?cited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs
  2. 2026Denser neq Better: Limits of On-Policy Self-Distillation for Continual Post-Training
  3. 2026Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention
  4. 2026Learning without Overwriting: A Theory of Self-Distillation and Supervised Fine-Tuning in Continual Reasoning
  5. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
  6. 2026A Predictive Law for On-Policy Self-Distillation From World Feedback
  7. 2026Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation
  8. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops