mlsanityData to 14 Oct 2026

Measuring Catastrophic Forgetting in Neural Networks

arXiv 1708.02072 ↗First posted 7 Aug 2017cs.AI cs.CV cs.LGcited by 360

Not on the map. Why?

Abstract (arXiv)

Deep neural networks are used in many state-of-the-art systems for machine perception. Once a network is trained to do a specific task, e.g., bird classification, it cannot easily be trained to do new tasks, e.g., incrementally learning to recognize additional bird species or learning an entirely different task such as flower recognition. When new tasks are added, typical deep neural networks are prone to catastrophically forgetting previous tasks. Networks that are capable of assimilating new information incrementally, much like how humans form new memories over time, will be more efficient than re-training the model from scratch each time a new task needs to be learned. There have been multiple attempts to develop schemes that mitigate catastrophic forgetting, but these methods have not been directly compared, the tests used to evaluate them vary considerably, and these methods have only been evaluated on small-scale problems (e.g., MNIST). In this paper, we introduce new metrics and benchmarks for directly comparing five different mechanisms designed to mitigate catastrophic forgetting in neural networks: regularization, ensembling, rehearsal, dual-memory, and sparse-coding. Our experiments on real-world images and sounds show that the mechanism(s) that are critical for optimal performance vary based on the incremental training paradigm and type of data being used, but they all demonstrate that the catastrophic forgetting problem has yet to be solved.

Key facts. Cited by 360 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Aug 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 75; 2022: 52; 2023: 64; 2024: 68; 2025: 65; 2026: 36.

360 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2024KAN: Kolmogorov-Arnold Networkscited by 649
  3. 2021Towards Personalized Federated Learningcited by 464
  4. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  5. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 245
  6. 2023Understanding plasticity in neural networkscited by 174
  7. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  8. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  9. 2023Large Language Models for Robotics: A Surveycited by 133
  10. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 131
  11. 2023Editing Implicit Assumptions in Text-to-Image Diffusion Modelscited by 130
  12. 2022FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learningcited by 121
38 more
  1. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 119
  2. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  3. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  4. 2021Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representationscited by 96
  5. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  6. 2024BadEdit: Backdooring large language models by model editingcited by 96
  7. 2022Source-Free Unsupervised Domain Adaptation: A Surveycited by 80
  8. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  9. 2021Replay in Deep Learning: Current Approaches and Missing Biological Elementscited by 72
  10. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  11. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  12. 2023TARGET: Federated Class-Continual Learning via Exemplar-Free Distillationcited by 65
  13. 2024Synthetic continued pretrainingcited by 60
  14. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 58
  15. 2021Self-Supervised Training Enhances Online Continual Learningcited by 54
  16. 2022A Closer Look at Rehearsal-Free Continual Learningcited by 54
  17. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
  18. 2021Non-Transferable Learning: A New Approach for Model Ownership Verification and Applicability Authorizationcited by 52
  19. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 50
  20. 2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 48
  21. 2021StyleAlign: Analysis and Applications of Aligned StyleGAN Modelscited by 48
  22. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  23. 2021FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learningcited by 44
  24. 2023GNNDelete: A General Strategy for Unlearning in Graph Neural Networkscited by 43
  25. 2021Provable Repair of Deep Neural Networkscited by 42
  26. 2021Graph Few-shot Class-incremental Learningcited by 41
  27. 2021DualNet: Continual Learning, Fast and Slowcited by 39
  28. 2024When Large Language Models Meet Vector Databases: A Surveycited by 39
  29. 2024A Survey of Mambacited by 39
  30. 2024Scaling Laws for Forgetting When Fine-Tuning Large Language Modelscited by 38
  31. 2021CoReD: Generalizing Fake Media Detection with Continual Representation using Distillationcited by 37
  32. 2021Non-IID data and Continual Learning processes in Federated Learning: A long road aheadcited by 36
  33. 2023The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Riskscited by 36
  34. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  35. 2023A Survey on Knowledge Editing of Neural Networkscited by 34
  36. 2021One Detector to Rule Them All: Towards a General Deepfake Attack Detection Frameworkcited by 33
  37. 2024Resurrecting Old Classes with New Data for Exemplar-Free Continual Learningcited by 33
  38. 2024Membership Inference Attacks against Large Vision-Language Modelscited by 33

50 most cited of 360 All 360 (JSON)