mlsanityData to 7 Oct 2026

Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent

arXiv 2006.11942 ↗First posted 21 Jun 2020stat.ML cs.LGcited by 68

Not on the map. Why?

Abstract (arXiv)

In Continual Learning settings, deep neural networks are prone to Catastrophic Forgetting. Orthogonal Gradient Descent was proposed to tackle the challenge. However, no theoretical guarantees have been proven yet. We present a theoretical framework to study Continual Learning algorithms in the Neural Tangent Kernel regime. This framework comprises closed form expression of the model through tasks and proxies for Transfer Learning, generalisation and tasks similarity. In this framework, we prove that OGD is robust to Catastrophic Forgetting then derive the first generalisation bound for SGD and OGD for Continual Learning. Finally, we study the limits of this framework in practice for OGD and highlight the importance of the Neural Tangent Kernel variation for Continual Learning with OGD.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 7; 2023: 13; 2024: 15; 2025: 10; 2026: 14.

68 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2021Gradient Projection Memory for Continual Learningcited by 338
  3. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  4. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  5. 2022A Theoretical Study on Solving Continual Learningcited by 78
  6. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  7. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  8. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  9. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 32
  10. 2023Continual Learning in Linear Classification on Separable Datacited by 31
  11. 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 28
  12. 2023Learnability and Algorithm for Continual Learningcited by 28
38 more
  1. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  2. 2024Understanding Forgetting in Continual Learning with Linear Regressioncited by 26
  3. 2021Formalizing the Generalization-Forgetting Trade-off in Continual Learningcited by 24
  4. 2023Fixed Design Analysis of Regularization-Based Continual Learningcited by 24
  5. 2021How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolationcited by 19
  6. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
  7. 2022Continual Learning Beyond a Single Modelcited by 18
  8. 2021Provable Lifelong Learning of Representationscited by 15
  9. 2021Provable Continual Learning via Sketched Jacobian Approximationscited by 15
  10. 2021Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgettingcited by 14
  11. 2022Provable and Efficient Continual Representation Learningcited by 14
  12. 2024Order parameters and phase transitions of continual learning in deep neural networkscited by 13
  13. 2024On the Diminishing Returns of Width for Continual Learningcited by 12
  14. 2022Selective Amnesia: On Efficient, High-Fidelity and Blind Suppression of Backdoor Effects in Trojaned Machine Learning Modelscited by 11
  15. 2024Last Iterate Convergence of Incremental Methods and Applications in Continual Learningcited by 10
  16. 2024Continual Memorization of Factoids in Language Modelscited by 8
  17. 2024NTK-Guided Few-Shot Class Incremental Learningcited by 7
  18. 2023Restricted Orthogonal Gradient Projection for Continual Learningcited by 6
  19. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
  20. 2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6
  21. 2026Self-Evolving Agents with Anytime-Valid Certificatescited by 6
  22. 2021TAG: Task-based Accumulated Gradients for Lifelong learningcited by 5
  23. 2025Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspectivecited by 5
  24. 2024A streamlined Approach to Multimodal Few-Shot Class Incremental Learning for Fine-Grained Datasetscited by 4
  25. 2025Memory-Statistics Tradeoff in Continual Learning with Structural Regularizationcited by 4
  26. 2024Data-dependent and Oracle Bounds on Forgetting in Continual Learningcited by 3
  27. 2025Global Convergence of Continual Learning on Non-IID Datacited by 3
  28. 2026Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretabilitycited by 3
  29. 2024Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspectivecited by 2
  30. 2025On the Theory of Continual Learning with Gradient Descent for Neural Networkscited by 2
  31. 2026Orthogonal Concept Erasure for Diffusion Modelscited by 2
  32. 2026Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptationcited by 2
  33. 2023SketchOGD: Memory-Efficient Continual Learningcited by 1
  34. 2024Buffer-based Gradient Projection for Continual Federated Learningcited by 1
  35. 2025Sample Compression for Self Certified Continual Learningcited by 1
  36. 2026Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformerscited by 1
  37. 2026Interference and Retention in Continual Learningcited by 1
  38. 2021Defeating Catastrophic Forgetting via Enhanced Orthogonal Weights Modificationcited by 0

50 most cited of 68 All 68 (JSON)