mlsanityData to 14 Oct 2026

Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual Learning

arXiv 2308.09544 ↗First posted 18 Aug 2023cs.LG cs.AI cs.CVcited by 16

On the map: Knowledge distillation → Distillation for continual learning

Abstract (arXiv)

In this work, we investigate exemplar-free class incremental learning (CIL) with knowledge distillation (KD) as a regularization strategy, aiming to prevent forgetting. KD-based methods are successfully used in CIL, but they often struggle to regularize the model without access to exemplars of the training data from previous tasks. Our analysis reveals that this issue originates from substantial representation shifts in the teacher network when dealing with out-of-distribution data. This causes large errors in the KD loss component, leading to performance degradation in CIL models. Inspired by recent test-time adaptation methods, we introduce Teacher Adaptation (TA), a method that concurrently updates the teacher and the main models during incremental training. Our method seamlessly integrates with KD-based CIL approaches and allows for consistent enhancement of their performance across multiple exemplar-free CIL benchmarks. The source code for our method is available at https://github.com/fszatkowski/cl-teacher-adaptation.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Distillation for continual learning, one of 12 directions in Knowledge distillation. Builds on 50 of its 70 references that are in the map. First posted on arXiv 18 Aug 2023.

Builds on

50 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022A Closer Look at Rehearsal-Free Continual Learning31 Mar

Other references · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2016Layer Normalizationcited by 5,669
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2016Learning without Forgettingcited by 2,496
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  10. 2019Searching for MobileNetV3cited by 1,994
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
37 more
  1. 2018Group Normalizationcited by 1,524
  2. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  3. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  4. 2018Experience Replay for Continual Learningcited by 1,095
  5. 2021Learning to Prompt for Continual Learningcited by 944
  6. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  8. 2019Large Scale Incremental Learningcited by 732
  9. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  10. 2019Three scenarios for continual learningcited by 614
  11. 2018End-to-End Incremental Learningcited by 597
  12. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 449
  13. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 431
  14. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 372
  15. 2018How Does Batch Normalization Help Optimization?cited by 357
  16. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  17. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 336
  18. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 283
  19. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  20. 2018Learning without Memorizingcited by 250
  21. 2019Class-incremental Learning via Deep Model Consolidationcited by 173
  22. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  23. 2020SS-IL: Separated Softmax for Incremental Learningcited by 126
  24. 2019Overcoming Catastrophic Forgetting with Unlabeled Data in the Wildcited by 111
  25. 2021PyCIL: A Python Toolbox for Class-Incremental Learningcited by 83
  26. 2020Memory Transformercited by 76
  27. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  28. 2021A study on the plasticity of neural networkscited by 44
  29. 2021Reproducibility Report: La-MAML: Look-ahead Meta Learning for Continual Learningcited by 42
  30. 2023Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learningcited by 41
  31. 2022Continual Normalization: Rethinking Batch Normalization for Online Continual Learningcited by 40
  32. 2022Parameter-Efficient Tuning on Layer Normalization for Pre-trained Language Modelscited by 19
  33. 2019DenseNet Models for Tiny ImageNet Classificationcited by 17
  34. 2022Rebalancing Batch Normalization for Exemplar-based Class-Incremental Learningcited by 16
  35. 2022A Memory Transformer Network for Incremental Learningcited by 9
  36. 2020Multi-level Knowledge Distillation via Knowledge Alignment and Correlationcited by 3
  37. 2023Improving Representational Continuity via Continued Pretrainingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 7; 2026: 1.

16 papers.

Sort:

By area:

  1. 2024Memory-efficient Continual Learning with Neural Collapse Contrastivecited by 8
  2. 2024Joint Diffusion models in Continual Learningcited by 6
  3. 2024Continual Test-Time Adaptation for Object Detection with Adaptive Monitoring and Randomized Restorationcited by 5
  4. 2024Not Just Object, But State: Compositional Incremental Learning without Forgettingcited by 2
  5. 2024Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Expertscited by 1
  6. 2024EchoDFKD: Data-Free Knowledge Distillation for Cardiac Ultrasound Segmentation using Synthetic Datacited by 1
  7. 2024Collaborative Knowledge Distillation via a Learning-by-Education Node Communitycited by 1
  8. 2025CIFNet: An Analytic Neural Learning Framework for Efficient and Calibrated Class-Incremental Learningcited by 1
  9. 2025CLAD-Net: Continual Activity Recognition in Multi-Sensor Wearable Systemscited by 1
  10. 2024ReFu: Recursive Fusion for Exemplar-Free 3D Class-Incremental Learningcited by 0
  11. 2025Learning Dynamic Representations via An Optimally-Weighted Maximum Mean Discrepancy Optimization Framework for Continual Learningcited by 0
  12. 2025Towards Robust Incremental Learning under Ambiguous Supervisioncited by 0
4 more
  1. 2025Audio-Visual Class-Incremental Learning for Fish Feeding intensity Assessment in Aquaculturecited by 0
  2. 2025MCIGLE: Multimodal Exemplar-Free Class-Incremental Graph Learningcited by 0
  3. 2025SCoDA: Self-supervised Continual Domain Adaptationcited by 0
  4. 2026Relative Parameter Importance in Task-Agnostic Replay-Free Continual Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Self-distilled Knowledge Delegator for Exemplar-free Class Incremental Learning
  2. 2025Low-Complexity Inference in Continual Learning via Compressed Knowledge Transfer
  3. 2025CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning
  4. 2021Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay
  5. 2024Continual Distillation Learning for Rehearsal-Free Class-Incremental Learning via Decoupled Prompting
  6. 2024Reducing Catastrophic Forgetting in Online Class Incremental Learning Using Self-Distillation
  7. 2025Multi-Level Knowledge Distillation and Dynamic Self-Supervised Learning for Continual Learning
  8. 2023Fine-Grained Knowledge Selection and Restoration for Non-Exemplar Class Incremental Learning