Data to 7 Oct 2026

A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix

arXiv 2010.04003 ↗First posted 7 Oct 2020cs.LG cs.AI stat.MLcited by 99

Not on the map. Why?

Abstract (arXiv)

Continual learning (CL) is a setting in which an agent has to learn from an incoming stream of data during its entire lifetime. Although major advances have been made in the field, one recurring problem which remains unsolved is that of Catastrophic Forgetting (CF). While the issue has been extensively studied empirically, little attention has been paid from a theoretical angle. In this paper, we show that the impact of CF increases as two tasks increasingly align. We introduce a measure of task similarity called the NTK overlap matrix which is at the core of CF. We analyze common projected gradient algorithms and demonstrate how they mitigate forgetting. Then, we propose a variant of Orthogonal Gradient Descent (OGD) which leverages structure of the data through Principal Component Analysis (PCA). Experiments support our theoretical findings and show how our method can help reduce CF on classical CL datasets.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 14; 2023: 17; 2024: 20; 2025: 16; 2026: 24.

99 papers.

Sort:

By area:

2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
2023Theory on Forgetting and Generalization of Continual Learningcited by 69
2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
2021Wide Neural Networks Forget Less Catastrophicallycited by 64
2023On the Stability-Plasticity Dilemma of Class-Incremental Learningcited by 55
2024Theory on Mixture-of-Experts in Continual Learningcited by 50
2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
2024A Statistical Theory of Regularization-Based Continual Learningcited by 32
2024On the Impact of Fine-Tuning on Chain-of-Thought Reasoningcited by 32
2023Continual Learning in Linear Classification on Separable Datacited by 31

38 more

2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 28
2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
2024Understanding Forgetting in Continual Learning with Linear Regressioncited by 26
2021Formalizing the Generalization-Forgetting Trade-off in Continual Learningcited by 24
2023Fixed Design Analysis of Regularization-Based Continual Learningcited by 24
2022Challenging Common Assumptions about Catastrophic Forgettingcited by 20
2021How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolationcited by 19
2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
2022Continual Learning Beyond a Single Modelcited by 18
2025Optimizers Qualitatively Alter Solutions And We Should Leverage Thiscited by 17
2022Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activationcited by 16
2021Provable Lifelong Learning of Representationscited by 15
2021Provable Continual Learning via Sketched Jacobian Approximationscited by 15
2023How connectivity structure shapes rich and lazy learning in neural circuitscited by 15
2021Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgettingcited by 14
2022Provable and Efficient Continual Representation Learningcited by 14
2023Hyp-OW: Exploiting Hierarchical Structure Learning with Hyperbolic Distance Enhances Open World Object Detectioncited by 14
2024Order parameters and phase transitions of continual learning in deep neural networkscited by 13
2024On the Diminishing Returns of Width for Continual Learningcited by 12
2022Beyond Supervised Continual Learning: a Reviewcited by 11
2022Selective Amnesia: On Efficient, High-Fidelity and Blind Suppression of Backdoor Effects in Trojaned Machine Learning Modelscited by 11
2024Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learningcited by 11
2024Last Iterate Convergence of Incremental Methods and Applications in Continual Learningcited by 10
2025From Continual Learning to SGD and Back: Better Rates for Continual Linear Modelscited by 10
2025Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regimecited by 10
2024What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinementcited by 9
2022Theoretical Understanding of the Information Flow on Continual Learning Performancecited by 8
2023Clustering-based Domain-Incremental Learningcited by 8
2024Demystifying Language Model Forgetting with Low-rank Example Associationscited by 8
2024LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learningcited by 8
2024Continual Memorization of Factoids in Language Modelscited by 8
2025Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspectivecited by 8
2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
2022Continual Feature Selection: Spurious Features in Continual Learningcited by 7
2024NTK-Guided Few-Shot Class Incremental Learningcited by 7
2023Efficient Parametric Approximations of Neural Network Function Space Distancecited by 6
2023Preserving Linear Separability in Continual Learning by Backward Feature Projectioncited by 6
2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6

50 most cited of 99 All 99 (JSON)