# A comprehensive, application-oriented study of catastrophic forgetting in DNNs

arXiv: https://arxiv.org/abs/1905.08101 · HTML: https://mlsanity.com/paper/1905.08101/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2019.

## Built on by (most cited first, 30 of 35)

- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Open-environment Machine Learning](https://mlsanity.com/paper/2206.00423/) (2022)
- [Continually Learning Self-Supervised Representations with Projected Functional Regularization](https://mlsanity.com/paper/2112.15022/) (2021)
- [On the Soft-Subnetwork for Few-shot Class Incremental Learning](https://mlsanity.com/paper/2209.07529/) (2022)
- [Online Class Incremental Learning on Stochastic Blurry Task Boundary via Mask and Visual Prompt Tuning](https://mlsanity.com/paper/2308.09303/) (2023)
- [Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regime](https://mlsanity.com/paper/2207.06475/) (2022)
- [CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability](https://mlsanity.com/paper/2110.03331/) (2021)
- [An Investigation of Replay-based Approaches for Continual Learning](https://mlsanity.com/paper/2108.06758/) (2021)
- [The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model](https://mlsanity.com/paper/2401.12617/) (2024)
- [Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy](https://mlsanity.com/paper/2312.10549/) (2023)
- [Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis](https://mlsanity.com/paper/2104.09396/) (2021)
- [RoRD: Rotation-Robust Descriptors and Orthographic Views for Local Feature Matching](https://mlsanity.com/paper/2103.08573/) (2021)
- [Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Models](https://mlsanity.com/paper/2305.09144/) (2023)
- [Towards Robust Graph Incremental Learning on Evolving Graphs](https://mlsanity.com/paper/2402.12987/) (2024)
- [Exploring System Performance of Continual Learning for Mobile and Embedded Sensing Applications](https://mlsanity.com/paper/2110.13290/) (2021)
- [Provable and Efficient Continual Representation Learning](https://mlsanity.com/paper/2203.02026/) (2022)
- [Beyond Supervised Continual Learning: a Review](https://mlsanity.com/paper/2208.14307/) (2022)
- [FastICARL: Fast Incremental Classifier and Representation Learning with Efficient Budget Allocation in Audio Sensing Applications](https://mlsanity.com/paper/2106.07268/) (2021)
- [Overcoming Catastrophic Forgetting with Gaussian Mixture Replay](https://mlsanity.com/paper/2104.09220/) (2021)
- [Predicting the Susceptibility of Examples to Catastrophic Forgetting](https://mlsanity.com/paper/2406.09935/) (2024)
- [Continual Learning with Fully Probabilistic Models](https://mlsanity.com/paper/2104.09240/) (2021)
- [A Procedural World Generation Framework for Systematic Evaluation of Continual Learning](https://mlsanity.com/paper/2106.02585/) (2021)
- [A Study of Continual Learning Methods for Q-Learning](https://mlsanity.com/paper/2206.03934/) (2022)
- [Adiabatic replay for continual learning](https://mlsanity.com/paper/2303.13157/) (2023)
- [VIPeR: Visual Incremental Place Recognition with Adaptive Mining and Continual Learning](https://mlsanity.com/paper/2407.21416/) (2024)
- [Reviewing continual learning from the perspective of human-level intelligence](https://mlsanity.com/paper/2111.11964/) (2021)
- [Benchmarking Continual Learning from Cognitive Perspectives](https://mlsanity.com/paper/2312.03309/) (2023)
- [Variational Bayes for Federated Continual Learning](https://mlsanity.com/paper/2405.14291/) (2024)
- [The Bayesian Approach to Continual Learning: An Overview](https://mlsanity.com/paper/2507.08922/) (2025)
- [Overcome Anterograde Forgetting with Cycled Memory Networks](https://mlsanity.com/paper/2112.02342/) (2021)

## Abstract (arXiv)

We present a large-scale empirical study of catastrophic forgetting (CF) in modern Deep Neural Network (DNN) models that perform sequential (or: incremental) learning. A new experimental protocol is proposed that enforces typical constraints encountered in application scenarios. As the investigation is empirical, we evaluate CF behavior on the hitherto largest number of visual classification datasets, from each of which we construct a representative number of Sequential Learning Tasks (SLTs) in close alignment to previous works on CF. Our results clearly indicate that there is no model that avoids CF for all investigated datasets and SLTs under application conditions. We conclude with a discussion of potential solutions and workarounds to CF, notably for the EWC and IMM models.