mlsanityData to 13 Oct 2026

The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model

arXiv 2401.12617 ↗First posted 23 Jan 2024cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

In continual learning, catastrophic forgetting is affected by multiple aspects of the tasks. Previous works have analyzed separately how forgetting is affected by either task similarity or overparameterization. In contrast, our paper examines how task similarity and overparameterization jointly affect forgetting in an analyzable model. Specifically, we focus on two-task continual linear regression, where the second task is a random orthogonal transformation of an arbitrary first task (an abstraction of random permutation tasks). We derive an exact analytical expression for the expected forgetting - and uncover a nuanced pattern. In highly overparameterized models, intermediate task similarity causes the most forgetting. However, near the interpolation threshold, forgetting decreases monotonically with the expected task similarity. We validate our findings with linear regression on synthetic data, and with neural networks on established permutation task benchmarks.

Key facts. Cited by 26 papers in the map (data through 13 Oct 2026). Builds on 18 of its 32 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

18 of 32 references.

References · 18

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,610
  2. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 718
  3. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  4. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 159
  5. 2018Towards Robust Evaluations of Continual Learningcited by 148
  6. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 100
  7. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 87
  8. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  9. 2022A Theoretical Study on Solving Continual Learningcited by 79
  10. 2023Theory on Forgetting and Generalization of Continual Learningcited by 70
  11. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  12. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
6 more
  1. 2019A comprehensive, application-oriented study of catastrophic forgetting in DNNscited by 35
  2. 2023Continual Learning in Linear Classification on Separable Datacited by 33
  3. 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 28
  4. 2023Fixed Design Analysis of Regularization-Based Continual Learningcited by 24
  5. 2021Provable Continual Learning via Sketched Jacobian Approximationscited by 15
  6. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 8; 2025: 11; 2026: 7.

26 papers.

Sort:

By area:

  1. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  2. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
  3. 2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
  4. 2025Optimal Task Order for Continual Learning of Multiple Taskscited by 14
  5. 2024Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learningcited by 11
  6. 2024Last Iterate Convergence of Incremental Methods and Applications in Continual Learningcited by 10
  7. 2025Lightspeed Geometric Dataset Distance via Sliced Optimal Transportcited by 10
  8. 2025From Continual Learning to SGD and Back: Better Rates for Continual Linear Modelscited by 10
  9. 2025Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regimecited by 10
  10. 2025Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspectivecited by 9
  11. 2024Demystifying Language Model Forgetting with Low-rank Example Associationscited by 8
  12. 2024LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learningcited by 8
14 more
  1. 2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
  2. 2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6
  3. 2026Denser neq Better: Limits of On-Policy Self-Distillation for Continual Post-Trainingcited by 6
  4. 2025Memory-Statistics Tradeoff in Continual Learning with Structural Regularizationcited by 5
  5. 2025Global Convergence of Continual Learning on Non-IID Datacited by 3
  6. 2024Statistical curriculum learning: An elimination algorithm achieving an oracle riskcited by 1
  7. 2025On Understanding of the Dynamics of Model Capacity in Continual Learningcited by 1
  8. 2026Trust Region Continual Learning as an Implicit Meta-Learnercited by 1
  9. 2024Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Modelcited by 0
  10. 2026From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learningcited by 0
  11. 2026Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weightscited by 0
  12. 2026Understanding Imbalanced Forgetting in Rehearsal-Based Class-Incremental Learningcited by 0
  13. 2026Convergence of Continual Learning in Homogeneous Deep Networkscited by 0
  14. 2026The Sequential Price of Continual Learningcited by 0