mlsanityData to 7 Oct 2026

Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA

arXiv 2601.22828 ↗First posted 30 Jan 2026cs.LG cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Continual learning (CL) in vision-language models (VLMs) faces significant challenges in improving task adaptation and avoiding catastrophic forgetting. Existing methods usually have heavy inference burden or rely on external knowledge, while Low-Rank Adaptation (LoRA) has shown potential in reducing these issues by enabling parameter-efficient tuning. However, considering directly using LoRA to alleviate the catastrophic forgetting problem is non-trivial, we introduce a novel framework that restructures a single LoRA module as a decomposable Rank-1 Expert Pool. Our method learns to dynamically compose a sparse, task-specific update by selecting from this expert pool, guided by the semantics of the [CLS] token. In addition, we propose an Activation-Guided Orthogonal (AGO) loss that orthogonalizes critical parts of LoRA weights across tasks. This sparse composition and orthogonalization enable fewer parameter updates, resulting in domain-aware learning while minimizing inter-task interference and maintaining downstream task performance. Extensive experiments across multiple settings demonstrate state-of-the-art results in all metrics, surpassing zero-shot upper bounds in generalization. Notably, it reduces trainable parameters by 96.7% compared to the baseline method, eliminating reliance on external datasets or task-ID discriminators. The merged LoRAs retain less weights and incur no inference latency, making our method computationally lightweight.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 48 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

28 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters18 Mar
  2. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2016Learning without Forgettingcited by 2,465
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  9. 2021Robust fine-tuning of zero-shot modelscited by 785
  10. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  11. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  12. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
14 more
  1. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  2. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  3. 2024Higher Layers Need More LoRA Expertscited by 101
  4. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  5. 2022CLIP model is an Efficient Continual Learnercited by 64
  6. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  7. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 59
  8. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  9. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 41
  10. 2018Continual Classification Learning Using Generative Modelscited by 28
  11. 2022Do Pre-trained Models Benefit Equally in Continual Learning?cited by 20
  12. 2024Is Parameter Collision Hindering Continual Learning in LLMs?cited by 15
  13. 2025OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuningcited by 14
  14. 2024Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environmentscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Duala: Dual-Level Alignment of Subjects and Stimuli for Cross-Subject fMRI Decodingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models
  2. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusion
  3. 2026Black-Box Continual Learning for Vision-Language Models
  4. 2025Continual Learning in Vision-Language Models via Aligned Model Merging
  5. 2026KeepLoRA: Continual Learning with Residual Gradient Adaptation
  6. 2026CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
  7. 2025Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models
  8. 2025LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning