mlsanityData to 14 Oct 2026

Adaptively Integrated Knowledge Distillation and Prediction Uncertainty for Continual Learning

arXiv 2301.07316 ↗First posted 18 Jan 2023cs.CV cs.LGcited by 0

On the map: Knowledge distillation → Distillation for continual learning

Abstract (arXiv)

Current deep learning models often suffer from catastrophic forgetting of old knowledge when continually learning new knowledge. Existing strategies to alleviate this issue often fix the trade-off between keeping old knowledge (stability) and learning new knowledge (plasticity). However, the stability-plasticity trade-off during continual learning may need to be dynamically changed for better model performance. In this paper, we propose two novel ways to adaptively balance model stability and plasticity. The first one is to adaptively integrate multiple levels of old knowledge and transfer it to each block level in the new model. The second one uses prediction uncertainty of old knowledge to naturally tune the importance of learning new knowledge during model training. To our best knowledge, this is the first time to connect model prediction uncertainty and knowledge distillation for continual learning. In addition, this paper applies a modified CutMix particularly to augment the data for old knowledge, further alleviating the catastrophic forgetting issue. Extensive evaluations on the CIFAR100 and the ImageNet datasets confirmed the effectiveness of the proposed method for continual learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Distillation for continual learning, one of 12 directions in Knowledge distillation. Builds on 34 of its 51 references that are in the map. First posted on arXiv 18 Jan 2023.

Builds on

34 of 51 references.

References · 34

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  2. 2017Mask R-CNNcited by 7,277
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  7. 2016Learning without Forgettingcited by 2,496
  8. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  10. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  11. 2017Continual Learning with Deep Generative Replaycited by 1,158
  12. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
22 more
  1. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  2. 2019Large Scale Incremental Learningcited by 732
  3. 2018End-to-End Incremental Learningcited by 597
  4. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  5. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  6. 2021Distilling Knowledge via Knowledge Reviewcited by 288
  7. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  8. 2021Co2L: Contrastive Continual Learningcited by 244
  9. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  10. 2020Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentationcited by 222
  11. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  12. 2020Remix: Rebalanced Mixupcited by 144
  13. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 131
  14. 2020SS-IL: Separated Softmax for Incremental Learningcited by 126
  15. 2021Distilling Causal Effect of Data in Class-Incremental Learningcited by 117
  16. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  17. 2020Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identificationcited by 50
  18. 2021Efficient Feature Transformations for Discriminative and Generative Continual Learningcited by 50
  19. 2019Learning What and Where to Transfercited by 43
  20. 2021Do Not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgettingcited by 12
  21. 2015Learning Multiple Tasks with Multilinear Relationship Networkscited by 11
  22. 2018Closed-Loop Memory GAN for Continual Learningcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Unified Framework for Continual Learning and Unlearning
  2. 2022A Closer Look at Rehearsal-Free Continual Learning
  3. 2022Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation
  4. 2025Continual Learning in Vision-Language Models via Aligned Model Merging
  5. 2024Densely Distilling Cumulative Knowledge for Continual Learning
  6. 2025Low-Complexity Inference in Continual Learning via Compressed Knowledge Transfer
  7. 2025The Future of Continual Learning in the Era of Foundation Models: Three Key Directions
  8. 2022An Investigation of the Combination of Rehearsal and Knowledge Distillation in Continual Learning for Spoken Language Understanding