mlsanityData to 7 Oct 2026

Do Not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting

arXiv 2102.01906 ↗First posted 3 Feb 2021cs.LG cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

One of the major limitations of deep learning models is that they face catastrophic forgetting in an incremental learning scenario. There have been several approaches proposed to tackle the problem of incremental learning. Most of these methods are based on knowledge distillation and do not adequately utilize the information provided by older task models, such as uncertainty estimation in predictions. The predictive uncertainty provides the distributional information can be applied to mitigate catastrophic forgetting in a deep learning framework. In the proposed work, we consider a Bayesian formulation to obtain the data and model uncertainties. We also incorporate self-attention framework to address the incremental learning problem. We define distillation losses in terms of aleatoric uncertainty and self-attention. In the proposed work, we investigate different ablation analyses on these losses. Furthermore, we are able to obtain better results in terms of accuracy on standard benchmarks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 3 Feb 2021.

Builds on

38 of 47 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2016Wide Residual Networkscited by 3,057
  8. 2016Learning without Forgettingcited by 2,465
  9. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  10. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  12. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
26 more
  1. 2017Continual Learning with Deep Generative Replaycited by 1,152
  2. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  3. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  4. 2018Self-Attention Generative Adversarial Networkscited by 809
  5. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  6. 2018End-to-End Incremental Learningcited by 596
  7. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  8. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  9. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 281
  10. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  11. 2018Learning without Memorizingcited by 249
  12. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  13. 2020Adversarial Continual Learningcited by 140
  14. 2019Overcoming Catastrophic Forgetting with Unlabeled Data in the Wildcited by 111
  15. 2018Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilizationcited by 104
  16. 2018Generative replay with feedback connections as a general strategy for continual learningcited by 103
  17. 2017Encoder Based Lifelong Learningcited by 102
  18. 2020iTAML: An Incremental Task-Agnostic Meta-learning Approachcited by 90
  19. 2018Memory Replay GANs: learning to generate images from new categories without forgettingcited by 76
  20. 2018Generative Models from the perspective of Continual Learningcited by 69
  21. 2020ScaIL: Classifier Weights Scaling for Class Incremental Learningcited by 47
  22. 2019Attending to Discriminative Certainty for Domain Adaptationcited by 37
  23. 2019U-CAM: Visual Explanation using Uncertainty based Class Activation Mapscited by 36
  24. 2018Incremental Classifier Learning with Generative Adversarial Networkscited by 32
  25. 2018Bayesian Incremental Learning for Deep Neural Networkscited by 6
  26. 2018SupportNet: solving catastrophic forgetting in class incremental learning with support datacited by 2

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 1; 2023: 3; 2024: 0; 2025: 4.

12 papers.

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  3. 2022InCloud: Incremental Learning for Point Cloud Place Recognitioncited by 22
  4. 2023Continual Named Entity Recognition without Catastrophic Forgettingcited by 11
  5. 2021Dataset Knowledge Transfer for Class-Incremental Learning without Memorycited by 8
  6. 2021Data Uncertainty Guided Noise-aware Preprocessing Of Fingerprintscited by 3
  7. 2025DevFD: Developmental Face Forgery Detection by Learning Shared and Orthogonal LoRA Subspacescited by 3
  8. 2021Sensor-invariant Fingerprint ROI Segmentation Using Recurrent Adversarial Learningcited by 2
  9. 2025Strategic Base Representation Learning via Feature Augmentations for Few-Shot Class Incremental Learningcited by 1
  10. 2025Exploring Stability-Plasticity Trade-offs for Continual Named Entity Recognitioncited by 1
  11. 2023Adaptively Integrated Knowledge Distillation and Prediction Uncertainty for Continual Learningcited by 0
  12. 2025A Conformal Predictive Measure for Assessing Catastrophic Forgettingcited by 0