mlsanityData to 7 Oct 2026

Continual Learning and Catastrophic Forgetting

arXiv 2403.05175 ↗First posted 8 Mar 2024cs.LG cs.AI cs.CVcited by 73

Not on the map. Why?

Abstract (arXiv)

This book chapter delves into the dynamics of continual learning, which is the process of incrementally learning from a non-stationary stream of data. Although continual learning is a natural skill for the human brain, it is very challenging for artificial neural networks. An important reason is that, when learning something new, these networks tend to quickly and drastically forget what they had learned before, a phenomenon known as catastrophic forgetting. Especially in the last decade, continual learning has become an extensively studied topic in deep learning. This book chapter reviews the insights that this field has generated.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 119 of its 177 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

119 of 177 references.

References · 119

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2016Learning without Forgettingcited by 2,465
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  9. 2016Matching Networks for One Shot Learningcited by 2,246
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  11. 2016Progressive Neural Networkscited by 1,332
  12. 2017Continual Learning with Deep Generative Replaycited by 1,152
38 more
  1. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  2. 2018Experience Replay for Continual Learningcited by 1,074
  3. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  4. 2021Learning to Prompt for Continual Learningcited by 938
  5. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  6. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  7. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  8. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  9. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  10. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  11. 2019Gradient based sample selection for online continual learningcited by 568
  12. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  13. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  14. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  15. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  16. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  17. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  18. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  19. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  20. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  21. 2021Gradient Projection Memory for Continual Learningcited by 338
  22. 2017Variational Continual Learningcited by 335
  23. 2019Continual learning with hypernetworkscited by 267
  24. 2020Semantic Drift Compensation for Class-Incremental Learningcited by 241
  25. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  26. 2020Supermasks in Superpositioncited by 223
  27. 2018Task-Free Continual Learningcited by 219
  28. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  29. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  30. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  31. 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 178
  32. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  33. 2019Continual Unsupervised Representation Learningcited by 147
  34. 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 144
  35. 2020Continual Prototype Evolution: Learning Online from Non-Stationary Data Streamscited by 143
  36. 2020A Neural Dirichlet Process Mixture Model for Task-Free Continual Learningcited by 133
  37. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 124
  38. 2018Robust Classification with Convolutional Prototype Learningcited by 122

50 most cited of 119 All 119 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 34; 2026: 29.

73 papers.

Sort:

By area:

  1. 2024LeanAgent: Lifelong Learning for Formal Theorem Provingcited by 34
  2. 2024Rehearsal-Free Modular and Compositional Continual Learning for Language Modelscited by 26
  3. 2025Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Modelscited by 16
  4. 2025CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Surveycited by 15
  5. 2024Expand and Compress: Exploring Tuning Principles for Continual Spatio-Temporal Graph Forecastingcited by 10
  6. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 10
  7. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  8. 2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
  9. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 7
  10. 2025TableGPT-R1: Advancing Tabular Reasoning Through Reinforcement Learningcited by 7
  11. 2025Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptationcited by 6
  12. 2025UniCoRN: Latent Diffusion-based Unified Controllable Image Restoration Network across Multiple Degradationscited by 6
38 more
  1. 2024SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Modelscited by 5
  2. 2025DETACH: Cross-domain Learning for Long-Horizon Tasks via Mixture of Disentangled Expertscited by 5
  3. 2026MSSR: Memory-Aware Adaptive Replay for Continual LLM Fine-Tuningcited by 4
  4. 2026Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agentscited by 4
  5. 2025Replay4NCL: An Efficient Memory Replay-based Methodology for Neuromorphic Continual Learning in Embedded AI Systemscited by 3
  6. 2025Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learningcited by 3
  7. 2025We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrongcited by 3
  8. 2024CLASSP: a Biologically-Inspired Approach to Continual Learning through Adjustment Suppression and Sparsity Promotioncited by 2
  9. 2024U-TELL: Unsupervised Task Expert Lifelong Learningcited by 2
  10. 2024Lifelong Learning of Video Diffusion Models From a Single Video Streamcited by 2
  11. 2025Foundations of a Developmental Design Paradigm for Integrated Continual Learning, Deliberative Behavior, and Comprehensibilitycited by 2
  12. 2025Exploring Incremental Unlearning: Techniques, Challenges, and Future Directionscited by 2
  13. 2025LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Modelscited by 2
  14. 2025Replay to Remember (R2R): An Efficient Uncertainty-driven Unsupervised Continual Learning Framework Using Generative Replaycited by 2
  15. 2025Named Entity Recognition in Historical Italian: The Case of Giacomo Leopardi's Zibaldonecited by 2
  16. 2025Spatial Audio Motion Understanding and Reasoningcited by 2
  17. 2025Distributed Learning and Inference Systems: A Networking Perspectivecited by 1
  18. 2025Neural Networks Remember More: The Power of Parameter Isolation and Combinationcited by 1
  19. 2025Privacy-Aware Lifelong Learningcited by 1
  20. 2025Continual Learning with Columnar Spiking Neural Networkscited by 1
  21. 2026Reinforced Fast Weights with Next-Sequence Predictioncited by 1
  22. 2026Learning in the Null Space: Small Singular Values for Continual Learningcited by 1
  23. 2026ARROW: Augmented Replay for RObust World modelscited by 1
  24. 2026Incremental learning for audio classification with Hebbian Deep Neural Networkscited by 1
  25. 2026From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Mergingcited by 1
  26. 2024One system for learning and remembering episodes and rulescited by 0
  27. 2024From Lab to Pocket: A Novel Continual Learning-based Mobile Application for Screening COVID-19cited by 0
  28. 2025Sparks of cognitive flexibility: self-guided context inference for flexible stimulus-response mapping by attentional routingcited by 0
  29. 2025Staying Alive: Online Neural Network Maintenance and Systemic Driftcited by 0
  30. 2025Audio-Visual Class-Incremental Learning for Fish Feeding intensity Assessment in Aquaculturecited by 0
  31. 2025Autonomous Cyber Resilience via a Co-Evolutionary Arms Race within a Fortified Digital Twin Sandboxcited by 0
  32. 2025The Endless Tuning. An Artificial Intelligence Design To Avoid Human Replacement and Trace Back Responsibilitiescited by 0
  33. 2025Forgetting of task-specific knowledge in model merging-based continual learningcited by 0
  34. 2025Prediction-Oriented Subsampling from Data Streamscited by 0
  35. 2025Generative KI f"ur TAcited by 0
  36. 2025An Arbitration Control for an Ensemble of Diversified DQN variants in Continual Reinforcement Learningcited by 0
  37. 2025Diagnosing Shortcut-Induced Rigidity in Continual Learning: The Einstellung Rigidity Index (ERI)cited by 0
  38. 2025ZeBROD: Zero-Retraining Based Recognition and Object Detection Frameworkcited by 0

50 most cited of 73 All 73 (JSON)