mlsanityData to 14 Oct 2026

Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regime

arXiv 2207.06475 ↗First posted 1 Jun 2022cs.LG cs.AI stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

Overparameterization is known to permit strong generalization performance in neural networks. In this work, we provide an initial theoretical analysis of its effect on catastrophic forgetting in a continual learning setup. We show experimentally that in permuted MNIST image classification tasks, the generalization performance of multilayer perceptrons trained by vanilla stochastic gradient descent can be improved by overparameterization, and the extent of the performance increase achieved by overparameterization is comparable to that of state-of-the-art continual learning algorithms. We provide a theoretical explanation of this effect by studying a qualitatively similar two-task linear regression problem, where each task is related by a random orthogonal transformation. We show that when a model is trained on the two tasks in sequence without any additional regularization, the risk gain on the first task is small if the model is sufficiently overparameterized.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 1 Jun 2022.

Builds on

21 of 29 references.

References · 21

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2016Learning without Forgettingcited by 2,496
  3. 2017Continual Learning with Deep Generative Replaycited by 1,158
  4. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  5. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  6. 2019Benign Overfitting in Linear Regressioncited by 549
  7. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  8. 2019Two models of double descent for weak featurescited by 195
  9. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
  10. 2018Towards Robust Evaluations of Continual Learningcited by 148
  11. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 103
  12. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 88
9 more
  1. 2020Optimal Continual Learning has Perfect Memory and is NP-hardcited by 87
  2. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 81
  3. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  4. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  5. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  6. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 57
  7. 2022Architecture Matters in Continual Learningcited by 46
  8. 2019A comprehensive, application-oriented study of catastrophic forgetting in DNNscited by 35
  9. 2021Provable Continual Learning via Sketched Jacobian Approximationscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 12; 2026: 8.

29 papers.

Sort:

By area:

  1. 2023Theory on Forgetting and Generalization of Continual Learningcited by 72
  2. 2023Continual Learning in Linear Classification on Separable Datacited by 34
  3. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  4. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
  5. 2025Optimal Task Order for Continual Learning of Multiple Taskscited by 14
  6. 2025From Continual Learning to SGD and Back: Better Rates for Continual Linear Modelscited by 12
  7. 2025Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regimecited by 12
  8. 2024Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learningcited by 11
  9. 2024Last Iterate Convergence of Incremental Methods and Applications in Continual Learningcited by 10
  10. 2025Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspectivecited by 10
  11. 2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
  12. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
17 more
  1. 2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6
  2. 2025Memory-Statistics Tradeoff in Continual Learning with Structural Regularizationcited by 5
  3. 2025Replay Can Provably Increase Forgettingcited by 5
  4. 2026Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretabilitycited by 4
  5. 2025On the Theory of Continual Learning with Gradient Descent for Neural Networkscited by 3
  6. 2024Statistical curriculum learning: An elimination algorithm achieving an oracle riskcited by 1
  7. 2024Effectiveness of Distributed Gradient Descent with Local Steps for Overparameterized Modelscited by 1
  8. 2025Sample Compression for Self Certified Continual Learningcited by 1
  9. 2025On Understanding of the Dynamics of Model Capacity in Continual Learningcited by 1
  10. 2026Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspectivecited by 1
  11. 2026Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weightscited by 1
  12. 2025Integrating Frequency-Domain Representations with Low-Rank Adaptation in Vision-Language Modelscited by 0
  13. 2026Understanding the Role of Rehearsal Scale in Continual Learning under Varying Model Capacitiescited by 0
  14. 2026From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learningcited by 0
  15. 2026Convergence of Continual Learning in Homogeneous Deep Networkscited by 0
  16. 2026The Sequential Price of Continual Learningcited by 0
  17. 2026Formalizing Forgetting: Theory of Forgetting for Online Continual Learningcited by 0