# KeepLoRA: Continual Learning with Residual Gradient Adaptation

arXiv: https://arxiv.org/abs/2601.19659 · HTML: https://mlsanity.com/paper/2601.19659/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- First posted on arXiv 27 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Built on by (most cited first, 21 of 21)

- [Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning](https://mlsanity.com/paper/2602.04998/) (2026)
- [Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation](https://mlsanity.com/paper/2602.00722/) (2026)
- [Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation](https://mlsanity.com/paper/2603.02224/) (2026)
- [PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective](https://mlsanity.com/paper/2605.28819/) (2026)
- [RL Forgets! Towards Continual Policy Optimization](https://mlsanity.com/paper/2607.04364/) (2026)
- [ACE-LoRA: Adaptive Orthogonal Decoupling for Continual Image Editing](https://mlsanity.com/paper/2605.14948/) (2026)
- [Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs](https://mlsanity.com/paper/2606.23063/) (2026)
- [Bilinear Optimization Divergence: Diagnosing Factor-Constrained LoRA Continual Learning](https://mlsanity.com/paper/2609.23594/) (2026)
- [Beyond Attention: True Adaptive World Models via Spherical Kernel Operator](https://mlsanity.com/paper/2603.13263/) (2026)
- [iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2605.19301/) (2026)
- [COVD: Continual Open-Vocabulary Object Detection with Novel Concept Injection](https://mlsanity.com/paper/2605.27116/) (2026)
- [Continual Learning in Transition](https://mlsanity.com/paper/2608.06216/) (2026)
- [OrthoSkillVLA: Continual Skill Learning via Gradient-Informed Skill Subspace Adaptation](https://mlsanity.com/paper/2608.19589/) (2026)
- [CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning](https://mlsanity.com/paper/2608.27867/) (2026)
- [Efficient Online Continual Foundation Model Fine-Tuning for Predictive Process Monitoring](https://mlsanity.com/paper/2608.28237/) (2026)
- [LoRA-RC: Reservoir Computing with Low-Rank Adaptation](https://mlsanity.com/paper/2609.12327/) (2026)
- [Past, Future, All at Once: Mitigating Stability-Plasticity Dilemma via Post-hoc JANUS Rectification](https://mlsanity.com/paper/2609.19985/) (2026)
- [Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning](https://mlsanity.com/paper/2609.36759/) (2026)
- [Stable-Plastic Subspace Construction for Differentially Private Continual LoRA](https://mlsanity.com/paper/2610.17904/) (2026)
- [Geometric Calibration of Supervised Fine-Tuning](https://mlsanity.com/paper/2610.18634/) (2026)
- [ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning](https://mlsanity.com/paper/2610.19700/) (2026)

## Abstract (arXiv)

Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances the three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA.