mlsanityData to 7 Oct 2026

Memory Aware Synapses: Learning what (not) to forget

arXiv 1711.09601 ↗First posted 27 Nov 2017cs.CV cs.AI stat.MLcited by 1,094

Not on the map. Why?

Cited by 44 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Humans can learn in a continuous manner. Old rarely utilized knowledge can be overwritten by new incoming information while important, frequently used knowledge is prevented from being erased. In artificial learning systems, lifelong learning so far has focused mainly on accumulating knowledge over tasks and overcoming catastrophic forgetting. In this paper, we argue that, given the limited model capacity and the unlimited new information to be learned, knowledge has to be preserved or erased selectively. Inspired by neuroplasticity, we propose a novel approach for lifelong learning, coined Memory Aware Synapses (MAS). It computes the importance of the parameters of a neural network in an unsupervised and online manner. Given a new sample which is fed to the network, MAS accumulates an importance measure for each parameter of the network, based on how sensitive the predicted output function is to a change in this parameter. When learning a new task, changes to important parameters can then be penalized, effectively preventing important knowledge related to previous tasks from being overwritten. Further, we show an interesting connection between a local version of our method and Hebb's rule,which is a model for the learning process in the brain. We test our method on a sequence of object recognition tasks and on the challenging problem of learning an embedding for predicting <subject, predicate, object> triplets. We show state-of-the-art performance and, for the first time, the ability to adapt the importance of the parameters based on unlabeled data towards what the network needs (not) to forget, which may vary depending on test conditions.

Key facts. Cited by 1,094 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 133; 2022: 152; 2023: 229; 2024: 229; 2025: 191; 2026: 160.

1,094 papers.

Sort:

By area:

  1. 2021Learning to Prompt for Continual Learningcited by 938
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  3. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  4. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  5. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  6. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  7. 2021Gradient Projection Memory for Continual Learningcited by 338
  8. 2021Towards Open World Object Detectioncited by 297
  9. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  10. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  11. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
  12. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 228
38 more
  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  2. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 215
  3. 2023Class-Incremental Learning: A Surveycited by 209
  4. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  5. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 189
  6. 2023Mitigating the Alignment Tax of RLHFcited by 181
  7. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 178
  8. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  9. 2021Self-Supervised Models are Continual Learnerscited by 149
  10. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
  11. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 139
  12. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  13. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  14. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 130
  15. 2022Forward Compatible Few-Shot Class-Incremental Learningcited by 125
  16. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 124
  17. 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
  18. 2022Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learningcited by 118
  19. 2021AutoNovel: Automatically Discovering and Learning Novel Visual Categoriescited by 116
  20. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 115
  21. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidationcited by 113
  22. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  23. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  24. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  25. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 90
  26. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  27. 2022Few-Shot Class-Incremental Learning by Sampling Multi-Phase Taskscited by 87
  28. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  29. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 80
  30. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  31. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
  32. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  33. 2021Rehearsal revealed: The limits and merits of revisiting samples in continual learningcited by 76
  34. 2022Probing Representation Forgetting in Supervised and Unsupervised Continual Learningcited by 74
  35. 2021AFEC: Active Forgetting of Negative Transfer in Continual Learningcited by 73
  36. 2024Continual Learning and Catastrophic Forgettingcited by 73
  37. 2022Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillationcited by 72
  38. 2021Replay in Deep Learning: Current Approaches and Missing Biological Elementscited by 71

50 most cited of 1,094 All 1,094 (JSON)