mlsanityData to 7 Oct 2026

How catastrophic can catastrophic forgetting be in linear regression?

arXiv 2205.09588 ↗First posted 19 May 2022cs.LG cs.NA math.NAcited by 80

Not on the map. Why?

Abstract (arXiv)

To better understand catastrophic forgetting, we study fitting an overparameterized linear model to a sequence of tasks with different input distributions. We analyze how much the model forgets the true labels of earlier tasks after training on subsequent tasks, obtaining exact expressions and bounds. We establish connections between continual learning in the linear setting and two other research areas: alternating projections and the Kaczmarz method. In specific settings, we highlight differences between forgetting and convergence to the offline solution as studied in those areas. In particular, when T tasks in d dimensions are presented cyclically for k iterations, we prove an upper bound of T^2 * min{1/sqrt(k), d/k} on the forgetting. This stands in contrast to the convergence to the offline solution, which can be arbitrarily slow according to existing alternating projection results. We further show that the T^2 factor can be lifted when tasks are presented in a random ordering.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 35 of its 83 references that are in the map. First posted on arXiv 19 May 2022.

Builds on

35 of 83 references.

References · 35

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2016Learning without Forgettingcited by 2,465
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  4. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  5. 2017Continual Learning with Deep Generative Replaycited by 1,152
  6. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  7. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  8. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  9. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  10. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  11. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  12. 2017Variational Continual Learningcited by 335
23 more
  1. 2017Three Factors Influencing Minima in SGDcited by 270
  2. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  3. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  4. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
  5. 2016Katyusha: The First Direct Acceleration of Stochastic Gradient Methodscited by 134
  6. 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 132
  7. 2019A continual learning survey: Defying forgetting in classification taskscited by 118
  8. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
  9. 2020Optimal Continual Learning has Perfect Memory and is NP-hardcited by 86
  10. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  11. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  12. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  13. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  14. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  15. 2016Tight Complexity Bounds for Optimizing Composite Objectivescited by 47
  16. 2021Last iterate convergence of SGD for Least-Squares in the Interpolation regimecited by 41
  17. 2020Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updatescited by 34
  18. 2018Theory of Curriculum Learning, with Convex Loss Functionscited by 33
  19. 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networkscited by 22
  20. 2021How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolationcited by 19
  21. 2020Stochastic Frank-Wolfe for Constrained Finite-Sum Minimizationcited by 14
  22. 2016Worst-case Complexity of Cyclic Coordinate Descent: O(n2) Gap with Randomized Versioncited by 6
  23. 2019Accelerated Sampling Kaczmarz Motzkin Algorithm for The Linear Feasibility Problemcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 10; 2024: 18; 2025: 23; 2026: 26.

80 papers.

Sort:

By area:

  1. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  2. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  3. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  4. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  5. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 32
  6. 2023Continual Learning in Linear Classification on Separable Datacited by 31
  7. 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 28
  8. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  9. 2024Understanding Forgetting in Continual Learning with Linear Regressioncited by 26
  10. 2023Fixed Design Analysis of Regularization-Based Continual Learningcited by 24
  11. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  12. 2022Challenging Common Assumptions about Catastrophic Forgettingcited by 20
38 more
  1. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
  2. 2025Continual Multimodal Contrastive Learningcited by 19
  3. 2025Optimal Task Order for Continual Learning of Multiple Taskscited by 14
  4. 2024Provable Contrastive Continual Learningcited by 13
  5. 2024Order parameters and phase transitions of continual learning in deep neural networkscited by 13
  6. 2024Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learningcited by 11
  7. 2024Last Iterate Convergence of Incremental Methods and Applications in Continual Learningcited by 10
  8. 2025From Continual Learning to SGD and Back: Better Rates for Continual Linear Modelscited by 10
  9. 2025Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regimecited by 10
  10. 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
  11. 2024What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinementcited by 9
  12. 2023Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Promptcited by 8
  13. 2024Demystifying Language Model Forgetting with Low-rank Example Associationscited by 8
  14. 2024LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learningcited by 8
  15. 2024Continual Memorization of Factoids in Language Modelscited by 8
  16. 2025Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspectivecited by 8
  17. 2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
  18. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 8
  19. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
  20. 2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6
  21. 2023Advancing continual lifelong learning in neural information retrieval: definition, dataset, framework, and empirical evaluationcited by 5
  22. 2024Theory of Mixture-of-Experts for Mobile Edge Computingcited by 5
  23. 2025Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspectivecited by 5
  24. 2025H2ST: Hierarchical Two-Sample Tests for Continual Out-of-Distribution Detectioncited by 4
  25. 2025Memory-Statistics Tradeoff in Continual Learning with Structural Regularizationcited by 4
  26. 2025Replay Can Provably Increase Forgettingcited by 4
  27. 2025Continual Hyperbolic Learning of Instances and Classescited by 4
  28. 2024Data-dependent and Oracle Bounds on Forgetting in Continual Learningcited by 3
  29. 2025Global Convergence of Continual Learning on Non-IID Datacited by 3
  30. 2025A statistical physics framework for optimal learningcited by 3
  31. 2026Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretabilitycited by 3
  32. 2025Information-Theoretic Generalization Bounds of Replay-based Continual Learningcited by 2
  33. 2025On the Theory of Continual Learning with Gradient Descent for Neural Networkscited by 2
  34. 2025Infinite-Dimensional Operator/Block Kaczmarz Algorithms: Regret Bounds and λ-Effectivenesscited by 2
  35. 2025Data Heterogeneity and Forgotten Labels in Split Federated Learningcited by 2
  36. 2026Last-Iterate Convergence of Randomized Kaczmarz and SGD with Greedy Step Sizecited by 2
  37. 2022Continual Learning with Distributed Optimization: Does CoCoA Forget?cited by 1
  38. 2024Effectiveness of Distributed Gradient Descent with Local Steps for Overparameterized Modelscited by 1

50 most cited of 80 All 80 (JSON)