mlsanityData to 7 Oct 2026

Continual Learning in Low-rank Orthogonal Subspaces

arXiv 2010.11635 ↗First posted 22 Oct 2020cs.LGcited by 99

Not on the map. Why?

Abstract (arXiv)

In continual learning (CL), a learner is faced with a sequence of tasks, arriving one after the other, and the goal is to remember all the tasks once the continual learning experience is finished. The prior art in CL uses episodic memory, parameter regularization or extensible network structures to reduce interference among tasks, but in the end, all the approaches learn different tasks in a joint vector space. We believe this invariably leads to interference among different tasks. We propose to learn tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Further, to keep the gradients of different tasks coming from these subspaces orthogonal to each other, we learn isometric mappings by posing network training as an optimization problem over the Stiefel manifold. To the best of our understanding, we report, for the first time, strong results over experience-replay baseline with and without memory on standard classification benchmarks in continual learning. The code is made publicly available.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 21; 2023: 24; 2024: 22; 2025: 17; 2026: 9.

99 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  3. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 224
  4. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  5. 2022A Theoretical Study on Solving Continual Learningcited by 78
  6. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  7. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  8. 2023TARGET: Federated Class-Continual Learning via Exemplar-Free Distillationcited by 64
  9. 2023Multi-Task Reinforcement Learning with Mixture of Orthogonal Expertscited by 45
  10. 2024ConFIG: Towards Conflict-free Training of Physics Informed Neural Networkscited by 43
  11. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  12. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
38 more
  1. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  2. 2022Continual Learning with Recursive Gradient Optimizationcited by 32
  3. 2024Visual Prompt Tuning in Null Space for Continual Learningcited by 30
  4. 2022Balancing Stability and Plasticity through Advanced Null Space in Continual Learningcited by 29
  5. 2023Learnability and Algorithm for Continual Learningcited by 28
  6. 2021Kernel Continual Learningcited by 27
  7. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  8. 2022A Multi-Head Model for Continual Learning via Out-of-Distribution Replaycited by 27
  9. 2021Formalizing the Generalization-Forgetting Trade-off in Continual Learningcited by 24
  10. 2022Continual Learning Based on OOD Detection and Task Maskingcited by 21
  11. 2023A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifoldcited by 21
  12. 2022One-shot Federated Learning without Server-side Trainingcited by 20
  13. 2024SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixturecited by 19
  14. 2021Continual Learning for Image-Based Camera Localizationcited by 18
  15. 2022Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activationscited by 18
  16. 2022Memory Bounds for Continual Learningcited by 18
  17. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  18. 2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
  19. 2023FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learningcited by 14
  20. 2023NPCL: Neural Processes for Uncertainty-Aware Continual Learningcited by 14
  21. 2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
  22. 2024Order parameters and phase transitions of continual learning in deep neural networkscited by 13
  23. 2022Modelling continual learning in humans with Hebbian context gating and exponentially decaying task signalscited by 12
  24. 2022Optimizing Class Distribution in Memory for Multi-Label Online Continual Learningcited by 12
  25. 2025Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Modelscited by 12
  26. 2026Mano: Restriking Manifold Optimization for LLM Trainingcited by 12
  27. 2022Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructionscited by 11
  28. 2024Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learningcited by 11
  29. 2024MuseumMaker: Continual Style Customization without Catastrophic Forgettingcited by 10
  30. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
  31. 2022Susceptibility of Continual Learning Against Adversarial Attackscited by 8
  32. 2022Incremental Task Learning with Incremental Rank Updatescited by 8
  33. 2023Clustering-based Domain-Incremental Learningcited by 8
  34. 2022Learning to Predict Gradients for Semi-Supervised Continual Learningcited by 7
  35. 2022Effects of Auxiliary Knowledge on Continual Learningcited by 7
  36. 2022SHELS: Exclusive Feature Sets for Novelty Detection and Continual Learning Without Class Boundariescited by 7
  37. 2024On the Convergence of Continual Learning with Adaptive Methodscited by 7
  38. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 7

50 most cited of 99 All 99 (JSON)