mlsanityData to 14 Oct 2026

KeepLoRA: Continual Learning with Residual Gradient Adaptation

arXiv 2601.19659 ↗First posted 27 Jan 2026cs.CV cs.LGcited by 21

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances the three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. First posted on arXiv 27 Jan 2026.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 21.

21 papers.

Sort:

By area:

  1. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  2. 2026Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptationcited by 5
  3. 2026Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptationcited by 4
  4. 2026PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspectivecited by 3
  5. 2026RL Forgets! Towards Continual Policy Optimizationcited by 2
  6. 2026ACE-LoRA: Adaptive Orthogonal Decoupling for Continual Image Editingcited by 1
  7. 2026Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMscited by 1
  8. 2026Bilinear Optimization Divergence: Diagnosing Factor-Constrained LoRA Continual Learningcited by 1
  9. 2026Beyond Attention: True Adaptive World Models via Spherical Kernel Operatorcited by 0
  10. 2026iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Modelscited by 0
  11. 2026COVD: Continual Open-Vocabulary Object Detection with Novel Concept Injectioncited by 0
  12. 2026Continual Learning in Transitioncited by 0
9 more
  1. 2026OrthoSkillVLA: Continual Skill Learning via Gradient-Informed Skill Subspace Adaptationcited by 0
  2. 2026CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuningcited by 0
  3. 2026Efficient Online Continual Foundation Model Fine-Tuning for Predictive Process Monitoringcited by 0
  4. 2026LoRA-RC: Reservoir Computing with Low-Rank Adaptationcited by 0
  5. 2026Past, Future, All at Once: Mitigating Stability-Plasticity Dilemma via Post-hoc JANUS Rectificationcited by 0
  6. 2026Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learningcited by 0
  7. 2026Stable-Plastic Subspace Construction for Differentially Private Continual LoRAcited by 0
  8. 2026Geometric Calibration of Supervised Fine-Tuningcited by 0
  9. 2026ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation
  2. 2026Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA
  3. 2025Continual Learning in Vision-Language Models via Aligned Model Merging
  4. 2026SAMBAR: Selective Anchoring via Method of Multipliers for Balanced Knowledge Acquisition and Retention in Vision-Language-Action Models
  5. 2026TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models
  6. 2025Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models
  7. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Models
  8. 2026Black-Box Continual Learning for Vision-Language Models