# Do Not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting

arXiv: https://arxiv.org/abs/2102.01906 · HTML: https://mlsanity.com/paper/2102.01906/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 47 references that are in the map.
- First posted on arXiv 3 Feb 2021.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [Learning without Memorizing](https://mlsanity.com/paper/1811.08051/) (2018)
- [Overcoming Catastrophic Forgetting by Incremental Moment Matching](https://mlsanity.com/paper/1703.08475/) (2017)
- [Adversarial Continual Learning](https://mlsanity.com/paper/2003.09553/) (2020)
- [Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild](https://mlsanity.com/paper/1903.12648/) (2019)
- [Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization](https://mlsanity.com/paper/1802.01569/) (2018)
- [Generative replay with feedback connections as a general strategy for continual learning](https://mlsanity.com/paper/1809.10635/) (2018)
- [Encoder Based Lifelong Learning](https://mlsanity.com/paper/1704.01920/) (2017)
- [iTAML: An Incremental Task-Agnostic Meta-learning Approach](https://mlsanity.com/paper/2003.11652/) (2020)

## Built on by (most cited first, 12 of 12)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [InCloud: Incremental Learning for Point Cloud Place Recognition](https://mlsanity.com/paper/2203.00807/) (2022)
- [Continual Named Entity Recognition without Catastrophic Forgetting](https://mlsanity.com/paper/2310.14541/) (2023)
- [Dataset Knowledge Transfer for Class-Incremental Learning without Memory](https://mlsanity.com/paper/2110.08421/) (2021)
- [Data Uncertainty Guided Noise-aware Preprocessing Of Fingerprints](https://mlsanity.com/paper/2107.01248/) (2021)
- [DevFD: Developmental Face Forgery Detection by Learning Shared and Orthogonal LoRA Subspaces](https://mlsanity.com/paper/2509.19230/) (2025)
- [Sensor-invariant Fingerprint ROI Segmentation Using Recurrent Adversarial Learning](https://mlsanity.com/paper/2107.01361/) (2021)
- [Strategic Base Representation Learning via Feature Augmentations for Few-Shot Class Incremental Learning](https://mlsanity.com/paper/2501.09361/) (2025)
- [Exploring Stability-Plasticity Trade-offs for Continual Named Entity Recognition](https://mlsanity.com/paper/2508.03259/) (2025)
- [Adaptively Integrated Knowledge Distillation and Prediction Uncertainty for Continual Learning](https://mlsanity.com/paper/2301.07316/) (2023)
- [A Conformal Predictive Measure for Assessing Catastrophic Forgetting](https://mlsanity.com/paper/2505.10677/) (2025)

## Abstract (arXiv)

One of the major limitations of deep learning models is that they face catastrophic forgetting in an incremental learning scenario. There have been several approaches proposed to tackle the problem of incremental learning. Most of these methods are based on knowledge distillation and do not adequately utilize the information provided by older task models, such as uncertainty estimation in predictions. The predictive uncertainty provides the distributional information can be applied to mitigate catastrophic forgetting in a deep learning framework. In the proposed work, we consider a Bayesian formulation to obtain the data and model uncertainties. We also incorporate self-attention framework to address the incremental learning problem. We define distillation losses in terms of aleatoric uncertainty and self-attention. In the proposed work, we investigate different ablation analyses on these losses. Furthermore, we are able to obtain better results in terms of accuracy on standard benchmarks.