mlsanityData to 14 Oct 2026

Distillation for continual learning

51 papers0 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: continual, forgetting, continual learning, catastrophic forgetting, catastrophic

Key facts. Distillation for continual learning: 51 papers on arXiv since 2021 (data through 14 Oct 2026). 7 of its 51 mapped papers (14%) cite “Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation” (2022, arXiv 2204.00895). 4 new papers 25 Sep–14 Oct.

What’s new

25 Sep – 14 Oct 2026

  1. Earlier · 4
  2. Continually Post-training Small Models via Synthetic Self-distillation also fits: On-policy and self-distillation for LLMs
  3. Learning without Overwriting: A Theory of Self-Distillation and Supervised Fine-Tuning in Continual Reasoning also fits: On-policy and self-distillation for LLMs
  4. Learning to Remember: Distilling Memory Retention for Compact Recurrent Neural Networks also fits: General knowledge distillation methods
  5. Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning

Foundations

Most cited by papers here

  1. Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation2022cited by 7 here
  2. On Learning the Geodesic Path for Incremental Learning2021cited by 5 here
  3. Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning2021cited by 3 here
  4. Class-Incremental Continual Learning into the eXtended DER-verse2022cited by 2 here
  5. Online Prototype Learning for Online Continual Learning2023cited by 2 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2021Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learningcited by 3 here
  2. 2021On Learning the Geodesic Path for Incremental Learningcited by 5 here
  3. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidationcited by 7 here
17 more it builds on, not in this chain
  1. 2021Distilling Knowledge via Knowledge Reviewcited by 2 here
  2. 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networkscited by 1 here
  3. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 1 here
  4. 2021FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learningcited by 1 here
  5. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 1 here
  6. 2021Knowledge distillation: A good teacher is patient and consistentcited by 1 here
  7. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 2 here
  8. 2022Consistent Representation Learning for Continual Relation Extractioncited by 1 here
  9. 2022Decoupled Knowledge Distillationcited by 2 here
  10. 2022A Closer Look at Rehearsal-Free Continual Learningcited by 1 here
  11. 2022Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replaycited by 1 here
  12. 2022Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentationcited by 1 here
  13. 2023SCLIFD:Supervised Contrastive Knowledge Distillation for Incremental Fault Diagnosis under Limited Fault Datacited by 1 here
  14. 2023Online Prototype Learning for Online Continual Learningcited by 2 here
  15. 2023MDCS: More Diverse Experts with Consistency Self-distillation for Long-tailed Recognitioncited by 1 here
  16. 2024Reducing Catastrophic Forgetting in Online Class Incremental Learning Using Self-Distillationcited by 1 here
  17. 2024DASK: Distribution Rehearsing via Adaptive Style Kernel Learning for Exemplar-Free Lifelong Person Re-Identificationcited by 1 here

Browse

30 of 51, ranked 6–35

  1. 6Reducing Catastrophic Forgetting in Online Class Incremental Learning Using Self-Distillation20241
  2. 7DASK: Distribution Rehearsing via Adaptive Style Kernel Learning for Exemplar-Free Lifelong Person Re-Identification20241
  3. 8SCLIFD:Supervised Contrastive Knowledge Distillation for Incremental Fault Diagnosis under Limited Fault Data20231
  4. 9Consistent Representation Learning for Continual Relation Extraction20221
  5. 10A Closer Look at Rehearsal-Free Continual Learning20221
  6. 11Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay20221
  7. 12Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation20221
  8. 13Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks20211
  9. 14SAE-FD: Sparse Autoencoder Feature Distillation for Continual Learning of Large Language Models20260
  10. 15Unsupervised Continual Clustering via Forward-Backward Knowledge Distillation20260
All 30
  1. 16NeuroGuard: Neural Gradient Update Aware of Representation Damage20260
  2. 17Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learning20260
  3. 18Class Incremental Fault Diagnosis under Limited Fault Data via Supervised Contrastive Knowledge Distillation20250
  4. 19S2IL: Structurally Stable Incremental Learning20250
  5. 20Distribution-aware Forgetting Compensation for Exemplar-Free Lifelong Person Re-identification20250
  6. 21Low-Complexity Inference in Continual Learning via Compressed Knowledge Transfer20250
  7. 22Prompt Tuning for Few-Shot Continual Learning Named Entity Recognition20250
  8. 23Multi-Level Knowledge Distillation and Dynamic Self-Supervised Learning for Continual Learning20250
  9. 24Knowledge Distillation for Continual Learning of Biomedical Neural Fields20250
  10. 25Scalable Class-Incremental Learning Based on Parametric Neural Collapse20250
  11. 26Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation20240
  12. 27Class-Incremental Few-Shot Event Detection20240
  13. 28Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners20240
  14. 29Brain-Inspired Continual Learning-Robust Feature Distillation and Re-Consolidation for Class Incremental Learning20240
  15. 30Densely Distilling Cumulative Knowledge for Continual Learning20240
  16. 31Forward-Backward Knowledge Distillation for Continual Clustering20240
  17. 32Decision Boundary-aware Knowledge Consolidation Generates Better Instance-Incremental Learner20240
  18. 33Continual Distillation Learning for Rehearsal-Free Class-Incremental Learning via Decoupled Prompting20240
  19. 34On Distilling the Displacement Knowledge for Few-Shot Class-Incremental Learning20240
  20. 35Effective Decision Boundary Learning for Class Incremental Learning20230