# Overcoming catastrophic forgetting in neural networks

arXiv: https://arxiv.org/abs/1612.00796 · HTML: https://mlsanity.com/paper/1612.00796/

## Key facts

- Cited by 4,787 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2016.

## Built on by (most cited first, 30 of 4787)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Continual Test-Time Domain Adaptation](https://mlsanity.com/paper/2203.13591/) (2022)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [Efficient Test-Time Model Adaptation without Forgetting](https://mlsanity.com/paper/2204.02610/) (2022)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)

## Abstract (arXiv)

The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Neural networks are not, in general, capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks which they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on the MNIST hand written digit dataset and by learning several Atari 2600 games sequentially.