mlsanityData to 7 Oct 2026

Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weighting

arXiv 2604.18075 ↗First posted 20 Apr 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

We investigate recently introduced domain-class incremental learning scenarios for vision-language models (VLMs). Recent works address this challenge using parameter-efficient methods, such as prefix-tuning or adapters, which facilitate model adaptation to downstream tasks by incorporating task-specific information into input tokens through additive vectors. However, previous approaches often normalize the weights of these vectors, disregarding the fact that different input tokens require different degrees of adjustment. To overcome this issue, we propose Dynamic Prefix Weighting (DPW), a framework that dynamically assigns weights to prefixes, complemented by adapters. DPW consists of 1) a gating module that adjusts the weights of each prefix based on the importance of the corresponding input token, and 2) a weighting mechanism that derives adapter output weights as a residual of prefix-tuning weights, ensuring that adapters are utilized only when necessary. Experimental results demonstrate that our method achieves state-of-the-art performance in domain-class incremental learning scenarios for VLMs. The code is available at: https://github.com/YonseiML/dpw.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 51 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

47 of 51 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusion12 Mar
  2. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Models6 Mar
  3. 2024Boosting Open-Domain Continual Learning via Leveraging Intra-domain Category-aware Prototype19 Aug
  4. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models7 Jul
  5. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters18 Mar
  6. 2024CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learning15 Mar
  7. 2024Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models14 Mar
  8. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar
  9. 2022Task Residual for Tuning Vision-Language Models18 Nov
  10. 2022Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model17 Aug
  11. 2022Conditional Prompt Learning for Vision-Language Models10 Mar

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  5. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  6. 2021Learning to Prompt for Continual Learningcited by 938
  7. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  8. 2021Robust fine-tuning of zero-shot modelscited by 785
  9. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  10. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  11. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  12. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
24 more
  1. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  2. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  3. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  4. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  5. 2018Learning without Memorizingcited by 249
  6. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  7. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  8. 2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 220
  9. 2018Task-Free Continual Learningcited by 219
  10. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 189
  11. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  12. 2024Continual Learning with Pre-Trained Models: A Surveycited by 151
  13. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
  14. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  15. 2024Theory, Analysis, and Best Practices for Sigmoid Self-Attentioncited by 62
  16. 2023Class-Incremental Exemplar Compression for Class-Incremental Learningcited by 50
  17. 2023On the Role of Attention in Prompt-tuningcited by 49
  18. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  19. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  20. 2022AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Taskscited by 29
  21. 2022Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learningcited by 27
  22. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
  23. 2024Dissecting Query-Key Interaction in Vision Transformerscited by 12
  24. 2024iVPT: Improving Task-relevant Information Sharing in Visual Prompt Tuning by Cross-layer Dynamic Connectioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Modular Embedding Recomposition for Incremental Learning
  2. 2025IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting
  3. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models
  4. 2023Class Incremental Learning with Pre-trained Vision-Language Models
  5. 2026Black-Box Continual Learning for Vision-Language Models
  6. 2024Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models
  7. 2026Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA
  8. 2026A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters