# The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model

arXiv: https://arxiv.org/abs/2401.12617 · HTML: https://mlsanity.com/paper/2401.12617/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 23 Jan 2024.

## Builds on (most cited first, 18 of 18)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics](https://mlsanity.com/paper/2007.07400/) (2020)
- [Towards Robust Evaluations of Continual Learning](https://mlsanity.com/paper/1805.09733/) (2018)
- [A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix](https://mlsanity.com/paper/2010.04003/) (2020)
- [Continual Learning in the Teacher-Student Setup: Impact of Task Similarity](https://mlsanity.com/paper/2107.04384/) (2021)
- [How catastrophic can catastrophic forgetting be in linear regression?](https://mlsanity.com/paper/2205.09588/) (2022)
- [A Theoretical Study on Solving Continual Learning](https://mlsanity.com/paper/2211.02633/) (2022)
- [Theory on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2302.05836/) (2023)
- [Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent](https://mlsanity.com/paper/2006.11942/) (2020)
- [Wide Neural Networks Forget Less Catastrophically](https://mlsanity.com/paper/2110.11526/) (2021)
- [A comprehensive, application-oriented study of catastrophic forgetting in DNNs](https://mlsanity.com/paper/1905.08101/) (2019)
- [Continual Learning in Linear Classification on Separable Data](https://mlsanity.com/paper/2306.03534/) (2023)
- [Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regime](https://mlsanity.com/paper/2207.06475/) (2022)
- [Fixed Design Analysis of Regularization-Based Continual Learning](https://mlsanity.com/paper/2303.10263/) (2023)
- [Provable Continual Learning via Sketched Jacobian Approximations](https://mlsanity.com/paper/2112.05095/) (2021)
- [On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspective](https://mlsanity.com/paper/2310.00137/) (2023)

## Built on by (most cited first, 26 of 26)

- [Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem](https://mlsanity.com/paper/2402.02868/) (2024)
- [Disentangling and Mitigating the Impact of Task Similarity for Continual Learning](https://mlsanity.com/paper/2405.20236/) (2024)
- [Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning](https://mlsanity.com/paper/2502.11019/) (2025)
- [Optimal Task Order for Continual Learning of Multiple Tasks](https://mlsanity.com/paper/2502.03350/) (2025)
- [From Continual Learning to SGD and Back: Better Rates for Continual Linear Models](https://mlsanity.com/paper/2504.04579/) (2025)
- [Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regime](https://mlsanity.com/paper/2507.11274/) (2025)
- [Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learning](https://mlsanity.com/paper/2408.16939/) (2024)
- [Last Iterate Convergence of Incremental Methods and Applications in Continual Learning](https://mlsanity.com/paper/2403.06873/) (2024)
- [Lightspeed Geometric Dataset Distance via Sliced Optimal Transport](https://mlsanity.com/paper/2501.18901/) (2025)
- [Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective](https://mlsanity.com/paper/2506.00205/) (2025)
- [Demystifying Language Model Forgetting with Low-rank Example Associations](https://mlsanity.com/paper/2406.14026/) (2024)
- [LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning](https://mlsanity.com/paper/2410.00645/) (2024)
- [Optimal Rates in Continual Linear Regression via Increasing Regularization](https://mlsanity.com/paper/2506.06501/) (2025)
- [Denser neq Better: Limits of On-Policy Self-Distillation for Continual Post-Training](https://mlsanity.com/paper/2607.01763/) (2026)
- [The Importance of Being Lazy: Scaling Limits of Continual Learning](https://mlsanity.com/paper/2506.16884/) (2025)
- [Memory-Statistics Tradeoff in Continual Learning with Structural Regularization](https://mlsanity.com/paper/2504.04039/) (2025)
- [Global Convergence of Continual Learning on Non-IID Data](https://mlsanity.com/paper/2503.18511/) (2025)
- [Statistical curriculum learning: An elimination algorithm achieving an oracle risk](https://mlsanity.com/paper/2402.13366/) (2024)
- [On Understanding of the Dynamics of Model Capacity in Continual Learning](https://mlsanity.com/paper/2508.08052/) (2025)
- [Trust Region Continual Learning as an Implicit Meta-Learner](https://mlsanity.com/paper/2602.02417/) (2026)
- [Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weights](https://mlsanity.com/paper/2604.17578/) (2026)
- [Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Model](https://mlsanity.com/paper/2410.16006/) (2024)
- [From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learning](https://mlsanity.com/paper/2604.13460/) (2026)
- [Understanding Imbalanced Forgetting in Rehearsal-Based Class-Incremental Learning](https://mlsanity.com/paper/2605.14785/) (2026)
- [Convergence of Continual Learning in Homogeneous Deep Networks](https://mlsanity.com/paper/2606.30559/) (2026)
- [The Sequential Price of Continual Learning](https://mlsanity.com/paper/2609.29674/) (2026)

## Abstract (arXiv)

In continual learning, catastrophic forgetting is affected by multiple aspects of the tasks. Previous works have analyzed separately how forgetting is affected by either task similarity or overparameterization. In contrast, our paper examines how task similarity and overparameterization jointly affect forgetting in an analyzable model. Specifically, we focus on two-task continual linear regression, where the second task is a random orthogonal transformation of an arbitrary first task (an abstraction of random permutation tasks). We derive an exact analytical expression for the expected forgetting - and uncover a nuanced pattern. In highly overparameterized models, intermediate task similarity causes the most forgetting. However, near the interpolation threshold, forgetting decreases monotonically with the expected task similarity. We validate our findings with linear regression on synthetic data, and with neural networks on established permutation task benchmarks.