mlsanityData to 7 Oct 2026

On the Diminishing Returns of Width for Continual Learning

arXiv 2403.06398 ↗First posted 11 Mar 2024cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

While deep neural networks have demonstrated groundbreaking performance in various settings, these models often suffer from catastrophic forgetting when trained on new tasks in sequence. Several works have empirically demonstrated that increasing the width of a neural network leads to a decrease in catastrophic forgetting but have yet to characterize the exact relationship between width and continual learning. We design one of the first frameworks to analyze Continual Learning Theory and prove that width is directly related to forgetting in Feed-Forward Networks (FFN). Specifically, we demonstrate that increasing network widths to reduce forgetting yields diminishing returns. We empirically verify our claims at widths hitherto unexplored in prior studies where the diminishing returns are clearly observed as predicted by our theory.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

33 of 48 references.

References · 33

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2016Wide Residual Networkscited by 3,057
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  5. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  6. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  8. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  9. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  10. 2018On Lazy Training in Differentiable Programmingcited by 671
  11. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
21 more
  1. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  2. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  3. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  4. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  5. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  6. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  7. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  8. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144
  9. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
  10. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  11. 2020Optimal Continual Learning has Perfect Memory and is NP-hardcited by 86
  12. 2022A Theoretical Study on Solving Continual Learningcited by 78
  13. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  14. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  15. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  16. 2019Generalization in Deep Networks: The Role of Distance from Initializationcited by 49
  17. 2022Architecture Matters in Continual Learningcited by 46
  18. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  19. 2019Continual Learning in Practicecited by 28
  20. 2021Knowledge-Adaptation Priorscited by 20
  21. 2023Efficient Parametric Approximations of Neural Network Function Space Distancecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 3.

12 papers.

  1. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  2. 2024Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learningcited by 11
  3. 2024LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learningcited by 8
  4. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  5. 2025Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspectivecited by 5
  6. 2026Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretabilitycited by 3
  7. 2025On the Theory of Continual Learning with Gradient Descent for Neural Networkscited by 2
  8. 2025Efficient Continual Learning through Frequency Decomposition and Integrationcited by 1
  9. 2025Rethinking Traffic Flow Forecasting: From Transition to Generatationcited by 0
  10. 2025What is the role of memorization in Continual Learning?cited by 0
  11. 2026Architecture-driven Shift: towards a lightweight selector for capturing the trends of logit shiftcited by 0
  12. 2026Convergence of Continual Learning in Homogeneous Deep Networkscited by 0