mlsanityData to 7 Oct 2026

Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models

arXiv 2403.09296 ↗First posted 14 Mar 2024cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large-scale vision-language models (VLMs) have shown a strong zero-shot generalization capability on unseen-domain data. However, adapting pre-trained VLMs to a sequence of downstream tasks often leads to the forgetting of previously learned knowledge and a reduction in zero-shot classification performance. To tackle this problem, we propose a unique Selective Dual-Teacher Knowledge Transfer framework that leverages the most recent fine-tuned and the original pre-trained VLMs as dual teachers to preserve the previously learned knowledge and zero-shot capabilities, respectively. With only access to an unlabeled reference dataset, our proposed framework performs a selective knowledge distillation mechanism by measuring the feature discrepancy from the dual-teacher VLMs. Consequently, our selective dual-teacher knowledge distillation mitigates catastrophic forgetting of previously learned knowledge while preserving the zero-shot capabilities of pre-trained VLMs. Extensive experiments on benchmark datasets demonstrate that our framework is favorable against state-of-the-art continual learning approaches for preventing catastrophic forgetting and zero-shot degradation. Project page: https://chuyu.org/research/snd

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 50 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

34 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2016Learning without Forgettingcited by 2,465
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
20 more
  1. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  2. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  4. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  5. 2021Learning to Prompt for Continual Learningcited by 938
  6. 2021Robust fine-tuning of zero-shot modelscited by 785
  7. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  8. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  9. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  10. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  11. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  12. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  13. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  14. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  15. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 130
  16. 2022Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replaycited by 70
  17. 2022R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learningcited by 59
  18. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  19. 2023AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learningcited by 46
  20. 2021Dual-Teacher Class-Incremental Learning With Data-Free Generative Replaycited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 7.

19 papers.

Sort:

By area:

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  2. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  3. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  4. 2024CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learningcited by 15
  5. 2025IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Promptingcited by 7
  6. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  7. 2025Affordance-First Decomposition for Continual Learning in Video-Language Understandingcited by 4
  8. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusioncited by 3
  9. 2026CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuningcited by 3
  10. 2025DesCLIP: Robust Continual Learning via General Attribute Descriptions for VLM-Based Visual Recognitioncited by 2
  11. 2025Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Modelscited by 1
  12. 2026Continual Learning with Vision-Language Models via Semantic-Geometry Preservationcited by 1
7 more
  1. 2026DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editingcited by 1
  2. 2025Rehearsal with Auxiliary-Informed Sampling for Audio Deepfake Detectioncited by 0
  3. 2025CaMiT: A Time-Aware Car Model Dataset for Classification and Generationcited by 0
  4. 2026Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weightingcited by 0
  5. 2026BESplit: Bias-Compensated Split Federated Learning with Evidential Aggregationcited by 0
  6. 2026Black-Box Continual Learning for Vision-Language Modelscited by 0
  7. 2026TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning
  2. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models
  3. 2025Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization
  4. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Models
  5. 2026SAMBAR: Selective Anchoring via Method of Multipliers for Balanced Knowledge Acquisition and Retention in Vision-Language-Action Models
  6. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation
  7. 2025Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models
  8. 2022Generative Negative Text Replay for Continual Vision-Language Pretraining