mlsanityData to 7 Oct 2026

Orthogonal Gradient Descent for Continual Learning

arXiv 1910.07104 ↗First posted 15 Oct 2019cs.LG stat.MLcited by 361

Not on the map. Why?

Abstract (arXiv)

Neural networks are achieving state of the art and sometimes super-human performance on learning tasks across a variety of domains. Whenever these problems require learning in a continual or sequential manner, however, neural networks suffer from the problem of catastrophic forgetting; they forget how to solve previous tasks after being trained on a new task, despite having the essential capacity to solve both tasks if they were trained on both simultaneously. In this paper, we propose to address this issue from a parameter space perspective and study an approach to restrict the direction of the gradient updates to avoid forgetting previously-learned data. We present the Orthogonal Gradient Descent (OGD) method, which accomplishes this goal by projecting the gradients from new tasks onto a subspace in which the neural network output on previous task does not change and the projected gradient is still in a useful direction for learning the new task. Our approach utilizes the high capacity of a neural network more efficiently and does not require storing the previously learned data that might raise privacy concerns. Experiments on common benchmarks reveal the effectiveness of the proposed OGD method.

Key facts. Cited by 361 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 43; 2023: 55; 2024: 68; 2025: 79; 2026: 89.

361 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  3. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
  4. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  5. 2021Gradient Projection Memory for Continual Learningcited by 338
  6. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  7. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  8. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  9. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 224
  10. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  11. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  12. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
38 more
  1. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  2. 2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 110
  3. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  4. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  5. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
  6. 2021Rehearsal revealed: The limits and merits of revisiting samples in continual learningcited by 76
  7. 2024Continual Learning and Catastrophic Forgettingcited by 73
  8. 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 71
  9. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  10. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  11. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  12. 2021Federated Learning with Fair Averagingcited by 59
  13. 2022Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfercited by 59
  14. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 59
  15. 2023Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimizationcited by 54
  16. 2021Model Zoo: A Growing "Brain" That Learns Continuallycited by 53
  17. 2023PCR: Proxy-based Contrastive Replay for Online Class-Incremental Continual Learningcited by 50
  18. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  19. 2022Architecture Matters in Continual Learningcited by 46
  20. 2022A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentialscited by 45
  21. 2024ConFIG: Towards Conflict-free Training of Physics Informed Neural Networkscited by 43
  22. 2023TiC-CLIP: Continual Training of CLIP Modelscited by 38
  23. 2024SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Modelscited by 37
  24. 2024Make Continual Learning Stronger via C-Flatcited by 35
  25. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  26. 2023Continual Learning with Scaled Gradient Projectioncited by 33
  27. 2024DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptationcited by 33
  28. 2021Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networkscited by 32
  29. 2022Continual Learning with Recursive Gradient Optimizationcited by 32
  30. 2024Convolutional Prompting meets Language Models for Continual Learningcited by 32
  31. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 32
  32. 2025Symmetry in Neural Network Parameter Spacescited by 31
  33. 2025Activation-Informed Merging of Large Language Modelscited by 29
  34. 2021Hierarchical Prototype Networks for Continual Graph Representation Learningcited by 28
  35. 2022Repairing Neural Networks by Leaving the Right Past Behindcited by 27
  36. 2023GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgettingcited by 27
  37. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  38. 2024Understanding Forgetting in Continual Learning with Linear Regressioncited by 26

50 most cited of 361 All 361 (JSON)