mlsanityData to 14 Oct 2026

An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks

arXiv 1312.6211 ↗First posted 21 Dec 2013stat.ML cs.LG cs.NEcited by 722

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Catastrophic forgetting is a problem faced by many machine learning models and algorithms. When trained on one task, then trained on a second task, many machine learning models "forget" how to perform the first task. This is widely believed to be a serious problem for neural networks. Here, we investigate the extent to which the catastrophic forgetting problem occurs for modern neural networks, comparing both established and recent gradient-based training algorithms and activation functions. We also examine the effect of the relationship between the first task and the second task on catastrophic forgetting. We find that it is always best to train using the dropout algorithm--the dropout algorithm is consistently best at adapting to the new task, remembering the old task, and has the best tradeoff curve between these two extremes. We find that different tasks and relationships between tasks result in very different rankings of activation function performance. This suggests the choice of activation function should always be cross-validated.

Key facts. Cited by 722 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Dec 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 98; 2022: 130; 2023: 129; 2024: 157; 2025: 114; 2026: 94.

722 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Modelscited by 909
  2. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  3. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  4. 2024LoRA Learns Less and Forgets Lesscited by 328
  5. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  6. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  7. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 245
  8. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  9. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  10. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  11. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
  12. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
38 more
  1. 2021Self-Supervised Models are Continual Learnerscited by 149
  2. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 148
  3. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  4. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 147
  5. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  6. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 144
  7. 2023Dataset Distillation: A Comprehensive Reviewcited by 127
  8. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  9. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 125
  10. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  11. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  12. 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
  13. 2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 117
  14. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 116
  15. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  16. 2021MetricGAN+: An Improved Version of MetricGAN for Speech Enhancementcited by 114
  17. 2023Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictorscited by 113
  18. 2022Dataset Condensation with Contrastive Signalscited by 105
  19. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 105
  20. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  21. 2023A Definition of Continual Reinforcement Learningcited by 101
  22. 2023Joint Audio and Speech Understandingcited by 100
  23. 2021Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representationscited by 96
  24. 2024BadEdit: Backdooring large language models by model editingcited by 96
  25. 2021Representational Continuity for Unsupervised Continual Learningcited by 95
  26. 2022Constrained Few-shot Class-incremental Learningcited by 94
  27. 2022The CLEAR Benchmark: Continual LEArning on Real-World Imagerycited by 93
  28. 2024Model Editing at Scale leads to Gradual and Catastrophic Forgettingcited by 93
  29. 2021Survey of Low-Resource Machine Translationcited by 92
  30. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 92
  31. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 90
  32. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 88
  33. 2022Few-Shot Class-Incremental Learning by Sampling Multi-Phase Taskscited by 88
  34. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  35. 2022Few-Shot Class-Incremental Learning from an Open-Set Perspectivecited by 84
  36. 2025Self-Adapting Language Modelscited by 84
  37. 2021RECALL: Replay-based Continual Learning in Semantic Segmentationcited by 83
  38. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 81

50 most cited of 722 All 722 (JSON)