mlsanityData to 14 Oct 2026

Wide Neural Networks Forget Less Catastrophically

arXiv 2110.11526 ↗First posted 21 Oct 2021cs.LG cs.AI cs.CVcited by 64

Not on the map. Why?

Abstract (arXiv)

A primary focus area in continual learning research is alleviating the "catastrophic forgetting" problem in neural networks by designing new algorithms that are more robust to the distribution shifts. While the recent progress in continual learning literature is encouraging, our understanding of what properties of neural networks contribute to catastrophic forgetting is still limited. To address this, instead of focusing on continual learning algorithms, in this work, we focus on the model itself and study the impact of "width" of the neural network architecture on catastrophic forgetting, and show that width has a surprisingly significant effect on forgetting. To explain this effect, we study the learning dynamics of the network from various perspectives such as gradient orthogonality, sparsity, and lazy training regime. We provide potential explanations that are consistent with the empirical results across different architectures and continual learning benchmarks.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Builds on 51 of its 69 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

51 of 69 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2016Wide Residual Networkscited by 3,070
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  9. 2016Learning without Forgettingcited by 2,496
  10. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  12. 2016Progressive Neural Networkscited by 1,361
38 more
  1. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  2. 2017Continual Learning with Deep Generative Replaycited by 1,158
  3. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  4. 2018Experience Replay for Continual Learningcited by 1,095
  5. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  7. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  8. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  9. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  10. 2018On Lazy Training in Differentiable Programmingcited by 675
  11. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  13. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  14. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 372
  15. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  16. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  17. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  18. 2017Variational Continual Learningcited by 339
  19. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  20. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  21. 2020Supermasks in Superpositioncited by 227
  22. 2019Meta-Learning Representations for Continual Learningcited by 189
  23. 2018Reinforced Continual Learningcited by 186
  24. 2020Understanding the Role of Training Regimes in Continual Learningcited by 180
  25. 2020Using Hindsight to Anchor Past Knowledge in Continual Learningcited by 161
  26. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 145
  27. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 126
  28. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 123
  29. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 103
  30. 2020Continual Learning in Low-rank Orthogonal Subspacescited by 99
  31. 2020Efficient Continual Learning with Modular Networks and Task-Driven Priorscited by 88
  32. 2020Learning to Continually Learncited by 84
  33. 2019Toward Understanding Catastrophic Forgetting in Continual Learningcited by 68
  34. 2018Modular meta-learningcited by 51
  35. 2018Automatically Composing Representation Transformations as a Means for Generalizationcited by 35
  36. 2021Dynamic Language Models for Continuously Evolving Contentcited by 27
  37. 2020The Effectiveness of Memory Replay in Large Scale Continual Learningcited by 26
  38. 2020Dropout as an Implicit Gating Mechanism For Continual Learningcited by 24

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 16; 2024: 14; 2025: 17; 2026: 10.

64 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  3. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  4. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  5. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 81
  6. 2022Architecture Matters in Continual Learningcited by 46
  7. 2022CoSCL: Cooperation of Small Continual Learners is Stronger than a Big Onecited by 43
  8. 2023Continual Learning as Computationally Constrained Reinforcement Learningcited by 37
  9. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 35
  10. 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 35
  11. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  12. 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 29
38 more
  1. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  2. 2023Learning and Forgetting Unsafe Examples in Large Language Modelscited by 24
  3. 2022Challenging Common Assumptions about Catastrophic Forgettingcited by 20
  4. 2023Is forgetting less a good inductive bias for forward transfer?cited by 18
  5. 2024Revisiting Neural Networks for Continual Learning: An Architectural Perspectivecited by 17
  6. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 17
  7. 2022Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activationcited by 16
  8. 2023Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgettingcited by 16
  9. 2025Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspectivecited by 14
  10. 2024On the Diminishing Returns of Width for Continual Learningcited by 12
  11. 2023Elephant Neural Networks: Born to Be a Continual Learnercited by 10
  12. 2023Continual Learning Under Language Shiftcited by 9
  13. 2024Demystifying Language Model Forgetting with Low-rank Example Associationscited by 8
  14. 2024PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learningcited by 8
  15. 2024LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learningcited by 8
  16. 2025Continual Learning in Vision-Language Models via Aligned Model Mergingcited by 8
  17. 2023Efficient Parametric Approximations of Neural Network Function Space Distancecited by 7
  18. 2023An Analysis of Initial Training Strategies for Exemplar-Free Class-Incremental Learningcited by 7
  19. 2024Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillationcited by 7
  20. 2025Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-trainingcited by 7
  21. 2023Heterogeneous Continual Learningcited by 6
  22. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
  23. 2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6
  24. 2025Active Learning for Continual Learning: Keeping the Past Alive in the Presentcited by 5
  25. 2025Continual Pre-training of MoEs: How robust is your router?cited by 5
  26. 2024Uncovering Latent Memories: Assessing Data Leakage and Memorization Patterns in Frontier AI Modelscited by 4
  27. 2024Continual learning with the neural tangent ensemblecited by 4
  28. 2025Analysis of Overparameterization in Continual Learning under a Linear Modelcited by 4
  29. 2026Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretabilitycited by 4
  30. 2025Online Continual Graph Learningcited by 3
  31. 2025On the Theory of Continual Learning with Gradient Descent for Neural Networkscited by 3
  32. 2023Mind the Backbone: Minimizing Backbone Distortion for Robust Object Detectioncited by 2
  33. 2024Improving Continual Learning Performance and Efficiency with Auxiliary Classifierscited by 2
  34. 2023Diagnosing Catastrophe: Large parts of accuracy loss in continual learning can be accounted for by readout misalignmentcited by 1
  35. 2024Overcoming Growth-Induced Forgetting in Task-Agnostic Continual Learningcited by 1
  36. 2025Achieving Deep Continual Learning via Evolutioncited by 1
  37. 2025Efficient Continual Learning through Frequency Decomposition and Integrationcited by 1
  38. 2025On Understanding of the Dynamics of Model Capacity in Continual Learningcited by 1

50 most cited of 64 All 64 (JSON)