# Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

arXiv: https://arxiv.org/abs/2603.00191 · HTML: https://mlsanity.com/paper/2603.00191/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 61 references that are in the map.
- First posted on arXiv 27 Feb 2026.

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Orthogonal Subspace Learning for Language Model Continual Learning](https://mlsanity.com/paper/2310.14152/) (2023)
- [Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models](https://mlsanity.com/paper/2305.12827/) (2023)
- [InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2404.00228/) (2024)
- [Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need](https://mlsanity.com/paper/2303.07338/) (2023)
- [SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model](https://mlsanity.com/paper/2303.05118/) (2023)
- [Task Singular Vectors: Reducing Task Interference in Model Merging](https://mlsanity.com/paper/2412.00081/) (2024)
- [Training Networks in Null Space of Feature Covariance for Continual Learning](https://mlsanity.com/paper/2103.07113/) (2021)
- [A Unified Continual Learning Framework with General Parameter-Efficient Tuning](https://mlsanity.com/paper/2303.10070/) (2023)
- [Model merging with SVD to tie the Knots](https://mlsanity.com/paper/2410.19735/) (2024)
- [Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality](https://mlsanity.com/paper/2310.07234/) (2023)
- [Learning without Forgetting for Vision-Language Models](https://mlsanity.com/paper/2305.19270/) (2023)
- [SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning](https://mlsanity.com/paper/2501.13198/) (2025)
- [On the Stability-Plasticity Dilemma of Class-Incremental Learning](https://mlsanity.com/paper/2304.01663/) (2023)
- [Consistent Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2403.08568/) (2024)
- [Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusion](https://mlsanity.com/paper/2407.14143/) (2024)

## Built on by (most cited first, 9 of 9)

- [Dual-Branch Cross-Projection Debiasing through Diffusion-based Disentanglement](https://mlsanity.com/paper/2606.24161/) (2026)
- [Symbiosis-Inspired Knowledge Distillation for Incremental Object Detection](https://mlsanity.com/paper/2607.13452/) (2026)
- [Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving](https://mlsanity.com/paper/2608.03579/) (2026)
- [PAPT++: Risk-Aware Adversarial Tuning and Generation for Single Domain Generalization](https://mlsanity.com/paper/2609.04837/) (2026)
- [CoGe-GCD: Reframing Generalized Category Discovery with Compositional Generalization](https://mlsanity.com/paper/2609.10158/) (2026)
- [CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework](https://mlsanity.com/paper/2609.17026/) (2026)
- [Distribution-Conditioned Task Routing for Class-Incremental Learning](https://mlsanity.com/paper/2609.34503/) (2026)
- [Generalized Biomedicine Discovery](https://mlsanity.com/paper/2610.00120/) (2026)
- [Stable-Plastic Subspace Construction for Differentially Private Continual LoRA](https://mlsanity.com/paper/2610.17904/) (2026)

## Abstract (arXiv)

Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods. Our code is available at https://github.com/HHHLF/LoDA_ICML2026.