mlsanityData to 14 Oct 2026

Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation

arXiv 2205.09029 ↗First posted 18 May 2022stat.ML cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Continual learning - learning new tasks in sequence while maintaining performance on old tasks - remains particularly challenging for artificial neural networks. Surprisingly, the amount of forgetting does not increase with the dissimilarity between the learned tasks, but appears to be worst in an intermediate similarity regime. In this paper we theoretically analyse both a synthetic teacher-student framework and a real data setup to provide an explanation of this phenomenon that we name Maslow's hammer hypothesis. Our analysis reveals the presence of a trade-off between node activation and node re-use that results in worst forgetting in the intermediate regime. Using this understanding we reinterpret popular algorithmic interventions for catastrophic interference in terms of this trade-off, and identify the regimes in which they are most effective.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 28 of its 59 references that are in the map. First posted on arXiv 18 May 2022.

Builds on

28 of 59 references.

References · 28

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2016Learning without Forgettingcited by 2,496
  5. 2016Progressive Neural Networkscited by 1,361
  6. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  7. 2017Continual Learning with Deep Generative Replaycited by 1,158
  8. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  9. 2018A Survey on Deep Transfer Learningcited by 419
  10. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  11. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
  12. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
16 more
  1. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 103
  2. 2020Efficient Continual Learning with Modular Networks and Task-Driven Priorscited by 88
  3. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 88
  4. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 69
  5. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  6. 2021Continual Learning via Local Module Compositioncited by 63
  7. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  8. 2020Lifelong Learning of Compositional Structurescited by 35
  9. 2016Neurogenesis Deep Learningcited by 32
  10. 2021An Analytical Theory of Curriculum Learning in Teacher-Student Networkscited by 27
  11. 2021Probing transfer learning with a model of synthetic correlated datasetscited by 25
  12. 2020Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysiscited by 23
  13. 2021Phase Transitions in Transfer Learning for High-Dimensional Perceptronscited by 22
  14. 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networkscited by 22
  15. 2021Algorithmic insights on continual learning from fruit fliescited by 11
  16. 2021Behavioral Experiments for Understanding Catastrophic Forgettingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 4; 2026: 5.

16 papers.

Sort:

By area:

  1. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
  2. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  3. 2023How connectivity structure shapes rich and lazy learning in neural circuitscited by 15
  4. 2024Order parameters and phase transitions of continual learning in deep neural networkscited by 14
  5. 2024Optimal Protocols for Continual Learning via Statistical Physics and Control Theorycited by 14
  6. 2025Forget Forgetting: Continual Learning in a World of Abundant Memorycited by 11
  7. 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
  8. 2025A Theory of Initialisation's Impact on Specialisationcited by 6
  9. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5
  10. 2025A statistical physics framework for optimal learningcited by 3
  11. 2025Make Haste Slowly: A Theory of Emergent Structured Mixed Selectivity in Feature Learning ReLU Networkscited by 2
  12. 2026Optimal Learning Rate Schedule for Balancing Effort and Performancecited by 1
4 more
  1. 2026Task-Induced Representational Invariances Depend on Learning Objective in Deep RLcited by 1
  2. 2026Continual Learning in Modern Hopfield Networks with an Application to Diffusion Modelscited by 0
  3. 2026Convergence of Continual Learning in Homogeneous Deep Networkscited by 0
  4. 2026A Dynamical Theory of LoRA in Continual Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Forgetting is Everywhere
  2. 2026Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretability
  3. 2026On the Plasticity Collapse in Continual Machine Unlearning
  4. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learning
  5. 2025Eidetic Learning: an Efficient and Provable Solution to Catastrophic Forgetting
  6. 2026Sparsity, Superposition, and Forgetting: A Mechanistic Study of Representation Retention in Continual Learning
  7. 2026Catastrophic Forgetting as Accessibility Collapse: A Three-Level Framework for Knowledge Persistence in Continual Learning
  8. 2024Remembering Transformer for Continual Learning