mlsanityData to 7 Oct 2026

A Methodology-Oriented Study of Catastrophic Forgetting in Incremental Deep Neural Networks

arXiv 2405.08015 ↗First posted 11 May 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Human being and different species of animals having the skills to gather, transferring knowledge, processing, fine-tune and generating information throughout their lifetime. The ability of learning throughout their lifespan is referred as continuous learning which is using neurocognition mechanism. Consequently, in real world computational system of incremental learning autonomous agents also needs such continuous learning mechanism which provide retrieval of information and long-term memory consolidation. However, the main challenge in artificial intelligence is that the incremental learning of the autonomous agent when new data confronted. In such scenarios, the main concern is catastrophic forgetting(CF), i.e., while learning the sequentially, neural network underfits the old data when it confronted with new data. To tackle this CF problem many numerous studied have been proposed, however it is very difficult to compare their performance due to dissimilarity in their evaluation mechanism. Here we focus on the comparison of all algorithms which are having similar type of evaluation mechanism. Here we are comparing three types of incremental learning methods: (1) Exemplar based methods, (2) Memory based methods, and (3) Network based method. In this survey paper, methodology oriented study for catastrophic forgetting in incremental deep neural network is addressed. Furthermore, it contains the mathematical overview of impact-full methods which can be help researchers to deal with CF.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 59 of its 76 references that are in the map. First posted on arXiv 11 May 2024.

Builds on

59 of 76 references.

References · 59

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2016Learning without Forgettingcited by 2,465
  7. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  9. 2016Progressive Neural Networkscited by 1,332
  10. 2017Continual Learning with Deep Generative Replaycited by 1,152
  11. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  12. 2018Experience Replay for Continual Learningcited by 1,074
38 more
  1. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  2. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  4. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  5. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  6. 2019Large Scale Incremental Learningcited by 727
  7. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  8. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  9. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  10. 2019Three scenarios for continual learningcited by 610
  11. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  12. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  13. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  14. 2017Variational Continual Learningcited by 335
  15. 2019On Tiny Episodic Memories in Continual Learningcited by 299
  16. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  17. 2018Task-Free Continual Learningcited by 219
  18. 2017Incremental Learning of Object Detectors without Catastrophic Forgettingcited by 208
  19. 2014Qualitatively characterizing neural network optimization problemscited by 186
  20. 2018Reinforced Continual Learningcited by 186
  21. 2017FearNet: Brain-Inspired Model for Incremental Learningcited by 177
  22. 2019Class-incremental Learning via Deep Model Consolidationcited by 172
  23. 2018Towards Robust Evaluations of Continual Learningcited by 148
  24. 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 144
  25. 2020Continual Prototype Evolution: Learning Online from Non-Stationary Data Streamscited by 143
  26. 2020Adversarial Continual Learningcited by 140
  27. 2018Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgettingcited by 127
  28. 2017Incremental Learning Through Deep Adaptationcited by 108
  29. 2020A Comprehensive Study of Class Incremental Learning Algorithms for Visual Taskscited by 107
  30. 2018Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilizationcited by 104
  31. 2017Encoder Based Lifelong Learningcited by 102
  32. 2016Less-forgetting Learning in Deep Neural Networkscited by 97
  33. 2020GAN Memory with No Forgettingcited by 84
  34. 2019Lifelong GAN: Continual Learning for Conditional Image Generationcited by 78
  35. 2018Task Agnostic Continual Learning Using Online Variational Bayescited by 61
  36. 2017Lifelong Generative Modelingcited by 51
  37. 2020Memory Efficient Class-Incremental Learning for Image Classificationcited by 50
  38. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 49

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.