# MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

arXiv: https://arxiv.org/abs/2610.02283 · HTML: https://mlsanity.com/paper/2610.02283/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Orthogonal Subspace Learning for Language Model Continual Learning](https://mlsanity.com/paper/2310.14152/) (2023)
- [InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2404.00228/) (2024)
- [Training Networks in Null Space of Feature Covariance for Continual Learning](https://mlsanity.com/paper/2103.07113/) (2021)
- [A Unified Continual Learning Framework with General Parameter-Efficient Tuning](https://mlsanity.com/paper/2303.10070/) (2023)
- [SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning](https://mlsanity.com/paper/2501.13198/) (2025)
- [Consistent Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2403.08568/) (2024)
- [Visual Prompt Tuning in Null Space for Continual Learning](https://mlsanity.com/paper/2406.05658/) (2024)
- [Dynamic Integration of Task-Specific Adapters for Class Incremental Learning](https://mlsanity.com/paper/2409.14983/) (2024)
- [Vector Quantization Prompting for Continual Learning](https://mlsanity.com/paper/2410.20444/) (2024)
- [SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting](https://mlsanity.com/paper/2505.22370/) (2025)
- [Continuous Subspace Optimization for Continual Learning](https://mlsanity.com/paper/2505.11816/) (2025)
- [Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning](https://mlsanity.com/paper/2502.07560/) (2025)
- [Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning](https://mlsanity.com/paper/2505.22389/) (2025)
- [PLAN: Proactive Low-Rank Allocation for Continual Learning](https://mlsanity.com/paper/2510.21188/) (2025)
- [EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learning](https://mlsanity.com/paper/2506.12351/) (2025)
- [CKAA: Cross-subspace Knowledge Alignment and Aggregation for Robust Continual Learning](https://mlsanity.com/paper/2507.09471/) (2025)
- [Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation](https://mlsanity.com/paper/2602.00722/) (2026)
- [Revisiting Weight Regularization for Low-Rank Continual Learning](https://mlsanity.com/paper/2602.17559/) (2026)

## Abstract (arXiv)

Low-rank adaptation (LoRA) provides a parameter-efficient approach to continual learning, but its nominal rank can conceal a loss of effective adaptation capacity. We identify spectral plasticity collapse: during sequential adaptation, update energy becomes concentrated in a small subset of singular modes, leaving much of the available low-rank space underutilized. This exposes a limitation of interference avoidance alone: protecting historical representations does not ensure that the remaining adaptation capacity is responsive to new tasks or effectively utilized. To address this problem, we propose textttMuLoRA, which jointly controls capacity allocation and utilization. First, historical whitening identifies input directions with strong current-task response relative to accumulated historical response, yielding a task-adaptive basis that remains fixed during training. Second, approximate polar orthogonalization of momentum updates reduces spectral concentration within theselected space. An orthonormal basis connects these mechanisms by transferring the factor-update spectrum exactly tothe induced weight update. We establish a max--min characterization of exact subspace selection and derive cumulative spectral bounds under controlled cross-step anisotropy. Across five class-incremental benchmarks and eight incremental settings, textttMuLoRA achieves the highest mean accuracy in 15 of 16 reported metrics.