mlsanityData to 14 Oct 2026

Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models

arXiv 2407.05342 ↗First posted 7 Jul 2024cs.CVcited by 34

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

This study addresses the Domain-Class Incremental Learning problem, a realistic but challenging continual learning scenario where both the domain distribution and target classes vary across tasks. To handle these diverse tasks, pre-trained Vision-Language Models (VLMs) are introduced for their strong generalizability. However, this incurs a new problem: the knowledge encoded in the pre-trained VLMs may be disturbed when adapting to new tasks, compromising their inherent zero-shot ability. Existing methods tackle it by tuning VLMs with knowledge distillation on extra datasets, which demands heavy computation overhead. To address this problem efficiently, we propose the Distribution-aware Interference-free Knowledge Integration (DIKI) framework, retaining pre-trained knowledge of VLMs from a perspective of avoiding information interference. Specifically, we design a fully residual mechanism to infuse newly learned knowledge into a frozen backbone, while introducing minimal adverse impacts on pre-trained knowledge. Besides, this residual property enables our distribution-aware integration calibration scheme, explicitly controlling the information implantation process for test data from unseen distributions. Experiments demonstrate that our DIKI surpasses the current state-of-the-art approach using only 0.86% of the trained parameters and requiring substantially less training time. Code is available at: https://github.com/lloongx/DIKI .

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 88 references that are in the map. First posted on arXiv 7 Jul 2024.

Builds on

73 of 88 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Learning without Forgetting for Vision-Language Models30 May
  2. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar
  3. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 68

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2016Learning without Forgettingcited by 2,496
  3. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  5. 2013Describing Textures in the Wildcited by 2,125
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  8. 2022Visual Prompt Tuningcited by 1,681
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  10. 2017Continual Learning with Deep Generative Replaycited by 1,158
  11. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  12. 2018Experience Replay for Continual Learningcited by 1,095
  13. 2021Learning to Prompt for Continual Learningcited by 944
  14. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  15. 2021Robust fine-tuning of zero-shot modelscited by 797
  16. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  17. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  18. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  19. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  20. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  21. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  22. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  23. 2019Three scenarios for continual learningcited by 614
  24. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  25. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  26. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  27. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  28. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  29. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  30. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  31. 2020Prior Guided Feature Enrichment Network for Few-Shot Segmentationcited by 271
  32. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  33. 2018Selective Experience Replay for Lifelong Learningcited by 241
  34. 2019Class-incremental Learning via Deep Model Consolidationcited by 173
  35. 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 146
  36. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  37. 2021Semi-supervised Semantic Segmentation with Directional Context-aware Consistencycited by 112
  38. 2023Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Groupingcited by 92

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 20; 2026: 7.

34 papers.

Sort:

By area:

  1. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  2. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 43
  3. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  4. 2024A Survey of Camouflaged Object Detection and Beyondcited by 37
  5. 2024Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediatorscited by 24
  6. 2025Does Your Vision-Language Model Get Lost in the Long Video Sampling Dilemma?cited by 20
  7. 2025Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Modelscited by 19
  8. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 18
  9. 2025MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Mergingcited by 14
  10. 2025LADA: Scalable Label-Specific CLIP Adapter for Continual Learningcited by 11
  11. 2025IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Promptingcited by 7
  12. 2024Typicalness-Aware Learning for Failure Detectioncited by 6
22 more
  1. 2025Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchorscited by 6
  2. 2025Sequence Transferability and Task Order Selection in Continual Learningcited by 5
  3. 2025Gamma: Toward Generic Image Assessment with Mixture of Assessment Expertscited by 5
  4. 2025Pay Less Attention to Deceptive Artifacts: Robust Detection of Compressed Deepfakes on Online Social Networkscited by 5
  5. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 5
  6. 2026Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptationcited by 5
  7. 2025Task-Core Memory Management and Consolidation for Long-term Continual Learningcited by 4
  8. 2025Affordance-First Decomposition for Continual Learning in Video-Language Understandingcited by 4
  9. 2025Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Modelscited by 3
  10. 2025iDPA: Instance Decoupled Prompt Attention for Incremental Medical Object Detectioncited by 3
  11. 2024Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learningcited by 2
  12. 2025Exploiting Discriminative Codebook Prior for Autoregressive Image Generationcited by 2
  13. 2024Retaining and Enhancing Pre-trained Knowledge in Vision-Language Models with Prompt Ensemblingcited by 1
  14. 2025Efficient Continual Learning through Frequency Decomposition and Integrationcited by 1
  15. 2025Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Modelscited by 1
  16. 2025Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Modelscited by 1
  17. 2026Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weightingcited by 0
  18. 2026Reasoning Portability: Guiding Continual Learning for MLLMs in the RLVR Eracited by 0
  19. 2026iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Modelscited by 0
  20. 2026COVD: Continual Open-Vocabulary Object Detection with Novel Concept Injectioncited by 0
  21. 2026KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptationcited by 0
  22. 2026Black-Box Continual Learning for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning
  2. 2024Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models
  3. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusion
  4. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Models
  5. 2025Continual Learning in Vision-Language Models via Aligned Model Merging
  6. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters
  7. 2025MLLM-CL: Continual Learning for Multimodal Large Language Models
  8. 2024Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models