mlsanityData to 14 Oct 2026

Overcoming catastrophic forgetting in neural networks

arXiv 1612.00796 ↗First posted 2 Dec 2016cs.LG cs.AI stat.MLcited by 4,787

Not on the map. Why?

Cited by 153 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Neural networks are not, in general, capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks which they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on the MNIST hand written digit dataset and by learning several Atari 2600 games sequentially.

Key facts. Cited by 4,787 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 491; 2022: 581; 2023: 772; 2024: 909; 2025: 924; 2026: 1110.

4,787 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Generative Adversarial Networkscited by 5,627
  4. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  5. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  7. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  8. 2021Learning to Prompt for Continual Learningcited by 944
  9. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  10. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  11. 2021Robust fine-tuning of zero-shot modelscited by 797
  12. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
38 more
  1. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  2. 2023ModelScope Text-to-Video Technical Reportcited by 658
  3. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  4. 2024KAN: Kolmogorov-Arnold Networkscited by 649
  5. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  6. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  7. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  8. 2021Editing Factual Knowledge in Language Modelscited by 553
  9. 2022Continual Test-Time Domain Adaptationcited by 525
  10. 2021Fast Model Editing at Scalecited by 498
  11. 2021iMAP: Implicit Mapping and Positioning in Real-Timecited by 476
  12. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 469
  13. 2021Towards Personalized Federated Learningcited by 464
  14. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  15. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  16. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  17. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  18. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  19. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  20. 2021Gradient Projection Memory for Continual Learningcited by 348
  21. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  22. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  23. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 324
  24. 2022A Survey of Machine Unlearningcited by 312
  25. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  26. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  27. 2021Towards Open World Object Detectioncited by 297
  28. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  29. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  30. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  31. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 271
  32. 2021Training Spiking Neural Networks Using Lessons From Deep Learningcited by 259
  33. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  34. 2026Self-Distillation Enables Continual Learningcited by 247
  35. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  36. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 245
  37. 2021Co2L: Contrastive Continual Learningcited by 244
  38. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238

50 most cited of 4,787 All 4,787 (JSON)