mlsanityData to 13 Oct 2026

Gradient Projection Memory for Continual Learning

arXiv 2103.09762 ↗First posted 17 Mar 2021cs.LG cs.CVcited by 344

Not on the map. Why?

Cited by 24 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability to learn continually without forgetting the past tasks is a desired attribute for artificial learning systems. Existing approaches to enable such learning in artificial neural networks usually rely on network growth, importance based weight update or replay of old data from the memory. In contrast, we propose a novel approach where a neural network learns new tasks by taking gradient steps in the orthogonal direction to the gradient subspaces deemed important for the past tasks. We find the bases of these subspaces by analyzing network representations (activations) after learning each task with Singular Value Decomposition (SVD) in a single shot manner and store them in the memory as Gradient Projection Memory (GPM). With qualitative and quantitative analyses, we show that such orthogonal gradient descent induces minimum to no interference with the past tasks, thereby mitigates forgetting. We evaluate our algorithm on diverse image classification datasets with short and long sequences of tasks and report better or on-par performance compared to the state-of-the-art approaches.

Key facts. Cited by 344 papers in the map (data through 13 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 17 Mar 2021.

Builds on

28 of 47 references.

References · 28

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,753
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,328
  3. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,323
  4. 2016Matching Networks for One Shot Learningcited by 2,251
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,882
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,666
  7. 2016Progressive Neural Networkscited by 1,347
  8. 2017Continual Learning with Deep Generative Replaycited by 1,155
  9. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,099
  10. 2018Efficient Lifelong Learning with A-GEMcited by 1,053
  11. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 891
  12. 2017Lifelong Learning with Dynamically Expandable Networkscited by 692
16 more
  1. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 675
  2. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 514
  3. 2019Orthogonal Gradient Descent for Continual Learningcited by 364
  4. 2019On Tiny Episodic Memories in Continual Learningcited by 303
  5. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  6. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  7. 2018Reinforced Continual Learningcited by 186
  8. 2018Towards Robust Evaluations of Continual Learningcited by 148
  9. 2020Adversarial Continual Learningcited by 140
  10. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  11. 2019Uncertainty-guided Continual Learning with Bayesian Neural Networkscited by 100
  12. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  13. 2017Deep Generative Dual Memory Network for Continual Learningcited by 59
  14. 2017Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharingcited by 22
  15. 2020SPACE: Structured Compression and Sharing of Representational Space for Continual Learningcited by 15
  16. 2018A Low Effort Approach to Structured CNN Design Using PCAcited by 14

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 14; 2022: 36; 2023: 53; 2024: 64; 2025: 74; 2026: 103.

344 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 862
  2. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 303
  3. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  4. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  5. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 232
  6. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 193
  7. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  8. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 81
  9. 2024Continual Learning and Catastrophic Forgettingcited by 74
  10. 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 71
  11. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 71
  12. 2023Theory on Forgetting and Generalization of Continual Learningcited by 70
38 more
  1. 2021Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inferencecited by 67
  2. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 67
  3. 2022Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfercited by 59
  4. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 59
  5. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  6. 2024Theory on Mixture-of-Experts in Continual Learningcited by 52
  7. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  8. 2021Natural continual learning: success is a journey, not (just) a destinationcited by 46
  9. 2022A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentialscited by 45
  10. 2022PIVOT: Prompting for Video Continual Learningcited by 45
  11. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  12. 2023Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projectioncited by 37
  13. 2024Make Continual Learning Stronger via C-Flatcited by 35
  14. 2023Continual Learning with Scaled Gradient Projectioncited by 34
  15. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  16. 2023A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithmcited by 33
  17. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 33
  18. 2022Continual Learning with Recursive Gradient Optimizationcited by 32
  19. 2024Convolutional Prompting meets Language Models for Continual Learningcited by 32
  20. 2024Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfercited by 31
  21. 2024Visual Prompt Tuning in Null Space for Continual Learningcited by 31
  22. 2022Improving Task-free Continual Learning by Distributionally Robust Memory Evolutioncited by 30
  23. 2022Balancing Stability and Plasticity through Advanced Null Space in Continual Learningcited by 30
  24. 2021Hierarchical Prototype Networks for Continual Graph Representation Learningcited by 28
  25. 2024Understanding Forgetting in Continual Learning with Linear Regressioncited by 28
  26. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  27. 2022A Multi-Head Model for Continual Learning via Out-of-Distribution Replaycited by 27
  28. 2023GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgettingcited by 27
  29. 2025CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Modelscited by 27
  30. 2022Meta Learning for Natural Language Processing: A Surveycited by 26
  31. 2021Optimizing Reusable Knowledge for Continual Learning via Metalearningcited by 24
  32. 2023Fixed Design Analysis of Regularization-Based Continual Learningcited by 24
  33. 2024Interactive Continual Learning: Fast and Slow Thinkingcited by 23
  34. 2024Unleashing the Power of Continual Learning on Non-Centralized Devices: A Surveycited by 23
  35. 2023Steering Prototypes with Prompt-tuning for Rehearsal-free Continual Learningcited by 22
  36. 2023Deep Unlearning: Fast and Efficient Gradient-free Approach to Class Forgettingcited by 22
  37. 2023Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospectscited by 22
  38. 2022Lifelong Reinforcement Learning with Modulating Maskscited by 21

50 most cited of 344 All 344 (JSON)