# Distillation for continual learning

Area: [Knowledge distillation](https://mlsanity.com/distill/index.md). HTML: https://mlsanity.com/distill/distillation-for-continual-learning/

## Key facts

- Distillation for continual learning: 51 papers on arXiv since 2021 (data through 14 Oct 2026).
- 7 of its 51 mapped papers (14%) cite “Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation” (2022, arXiv 2204.00895).
- 5 of its 51 mapped papers (10%) cite “On Learning the Geodesic Path for Incremental Learning” (2021, arXiv 2104.08572).
- 4 new papers 25 Sep–14 Oct.

## Most cited here

- [Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation](https://mlsanity.com/paper/2204.00895/) (2022), cited by 7 papers here
- [On Learning the Geodesic Path for Incremental Learning](https://mlsanity.com/paper/2104.08572/) (2021), cited by 5 papers here
- [Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2103.04059/) (2021), cited by 3 papers here
- [Class-Incremental Continual Learning into the eXtended DER-verse](https://mlsanity.com/paper/2201.00766/) (2022), cited by 2 papers here
- [Online Prototype Learning for Online Continual Learning](https://mlsanity.com/paper/2308.00301/) (2023), cited by 2 papers here
- [Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks](https://mlsanity.com/paper/2105.07385/) (2021), cited by 1 papers here
- [Consistent Representation Learning for Continual Relation Extraction](https://mlsanity.com/paper/2203.02721/) (2022), cited by 1 papers here
- [A Closer Look at Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2203.17269/) (2022), cited by 1 papers here
- [Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay](https://mlsanity.com/paper/2207.11213/) (2022), cited by 1 papers here
- [Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation](https://mlsanity.com/paper/2209.06107/) (2022), cited by 1 papers here

## New (25 Sep–14 Oct)

- [Continually Post-training Small Models via Synthetic Self-distillation](https://mlsanity.com/paper/2610.13113/) (2026-10-09)
- [Learning without Overwriting: A Theory of Self-Distillation and Supervised Fine-Tuning in Continual Reasoning](https://mlsanity.com/paper/2610.05200/) (2026-10-04)
- [Learning to Remember: Distilling Memory Retention for Compact Recurrent Neural Networks](https://mlsanity.com/paper/2610.06942/) (2026-10-03)
- [Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2609.39390/) (2026-09-30)