# Overcoming catastrophic forgetting in neural networks

arXiv: https://arxiv.org/abs/2507.10485 · HTML: https://mlsanity.com/paper/2507.10485/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 3 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Builds on (most cited first, 2 of 2)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)

## Built on by (most cited first, 30 of 55)

- [Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting](https://mlsanity.com/paper/2508.04227/) (2025)
- [Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning](https://mlsanity.com/paper/2509.08255/) (2025)
- [Forgetful but Faithful: A Cognitive Memory Architecture and Benchmark for Privacy-Aware Generative Agents](https://mlsanity.com/paper/2512.12856/) (2025)
- [AIMMerging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learning](https://mlsanity.com/paper/2509.17348/) (2025)
- [SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval](https://mlsanity.com/paper/2603.15599/) (2026)
- [Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training](https://mlsanity.com/paper/2605.09608/) (2026)
- [Weight Decay Improves Language Model Plasticity](https://mlsanity.com/paper/2602.11137/) (2026)
- [Diagnosing the Performance Trade-off in Moral Alignment: A Case Study on Gender Stereotypes](https://mlsanity.com/paper/2509.21456/) (2025)
- [Pruning as Regularization: Sensitivity-Aware One-Shot Pruning in ASR](https://mlsanity.com/paper/2511.08092/) (2025)
- [MedCL-Bench: Benchmarking stability-efficiency trade-offs and scaling in biomedical continual learning](https://mlsanity.com/paper/2603.16738/) (2026)
- [KEEP: Integrating Medical Ontologies with Clinical Data for Robust Code Embeddings](https://mlsanity.com/paper/2510.05049/) (2025)
- [Privacy-Aware Continual Self-Supervised Learning on Multi-Window Chest Computed Tomography for Domain-Shift Robustness](https://mlsanity.com/paper/2510.27213/) (2025)
- [Sparse Adapter Fusion for Continual Learning in NLP](https://mlsanity.com/paper/2602.02502/) (2026)
- [Learning Hierarchical Orthogonal Prototypes for Generalized Few-Shot 3D Point Cloud Segmentation](https://mlsanity.com/paper/2603.19788/) (2026)
- [Brainstacks: Cross-Domain Cognitive Capabilities via Frozen MoE-LoRA Stacks for Continual LLM Learning](https://mlsanity.com/paper/2604.01152/) (2026)
- [SCNO: Spiking Compositional Neural Operator -- Towards a Neuromorphic Foundation Model for Nuclear PDE Solving](https://mlsanity.com/paper/2604.11625/) (2026)
- [How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them](https://mlsanity.com/paper/2604.17105/) (2026)
- [Same Methods, Different Rankings: Trainable Depth as an Evaluation Variable in Continual Learning](https://mlsanity.com/paper/2604.21927/) (2026)
- [CMI-Mem: Toward Generalizable Long-Term Memory Management via CMI-Augmented Reinforcement Learning](https://mlsanity.com/paper/2607.20553/) (2026)
- [Verifier-Induced Support Reshaping in On-Policy Optimization](https://mlsanity.com/paper/2608.00220/) (2026)
- [Learning to Adapt Cross-Domain Preferences via Meta-LoRA for LLM Personalization](https://mlsanity.com/paper/2608.12389/) (2026)
- [AIM: Anchor Identity Features, Then Match for Multimodal Large Language Model Unlearning](https://mlsanity.com/paper/2608.28312/) (2026)
- [Where Identity Lives: Localized, Retain-Free Identity Unlearning in Multimodal Large Language Models](https://mlsanity.com/paper/2608.30649/) (2026)
- [Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgetting](https://mlsanity.com/paper/2508.05059/) (2025)
- [Investigation of D-Wave quantum annealing for training Restricted Boltzmann Machines and mitigating catastrophic forgetting](https://mlsanity.com/paper/2508.15697/) (2025)
- [BRAIN: Bias-Mitigation Continual Learning Approach to Vision-Brain Understanding](https://mlsanity.com/paper/2508.18187/) (2025)
- [Experience Scaling: Post-Deployment Evolution For Large Language Models](https://mlsanity.com/paper/2509.18771/) (2025)
- [On the Occurence of Critical Learning Periods in Neural Networks](https://mlsanity.com/paper/2510.09687/) (2025)
- [Adaptive transfer learning for surgical tool presence detection in laparoscopic videos through gradual freezing fine-tuning](https://mlsanity.com/paper/2510.15372/) (2025)
- [ProDER: A Continual Learning Approach for Fault Classification and Localization in Evolving Smart Grids](https://mlsanity.com/paper/2511.05420/) (2025)

## Abstract (arXiv)

Catastrophic forgetting is the primary challenge that hinders continual learning, which refers to a neural network ability to sequentially learn multiple tasks while retaining previously acquired knowledge. Elastic Weight Consolidation, a regularization-based approach inspired by synaptic consolidation in biological neural systems, has been used to overcome this problem. In this study prior research is replicated and extended by evaluating EWC in supervised learning settings using the PermutedMNIST and RotatedMNIST benchmarks. Through systematic comparisons with L2 regularization and stochastic gradient descent (SGD) without regularization, we analyze how different approaches balance knowledge retention and adaptability. Our results confirm what was shown in previous research, showing that EWC significantly reduces forgetting compared to naive training while slightly compromising learning efficiency on new tasks. Moreover, we investigate the impact of dropout regularization and varying hyperparameters, offering insights into the generalization of EWC across diverse learning scenarios. These results underscore EWC's potential as a viable solution for lifelong learning in neural networks.