mlsanityData to 7 Oct 2026

CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learning

arXiv 2403.10245 ↗First posted 15 Mar 2024cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

This paper explores the problem of continual learning (CL) of vision-language models (VLMs) in open domains, where the models need to perform continual updating and inference on a streaming of datasets from diverse seen and unseen domains with novel classes. Such a capability is crucial for various applications in open environments, e.g., AI assistants, autonomous driving systems, and robotics. Current CL studies mostly focus on closed-set scenarios in a single domain with known classes. Large pre-trained VLMs like CLIP have demonstrated superior zero-shot recognition ability, and a number of recent studies leverage this ability to mitigate catastrophic forgetting in CL, but they focus on closed-set CL in a single domain dataset. Open-domain CL of large VLMs is significantly more challenging due to 1) large class correlations and domain gaps across the datasets and 2) the forgetting of zero-shot knowledge in the pre-trained VLMs in addition to the knowledge learned from the newly adapted datasets. In this work we introduce a novel approach, termed CoLeCLIP, that learns an open-domain CL model based on CLIP. It addresses these challenges by a joint learning of a set of task prompts and a cross-domain class vocabulary. Extensive experiments on 11 domain datasets show that CoLeCLIP outperforms state-of-the-art methods for open-domain CL under both task- and class-incremental learning settings.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 61 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

36 of 61 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models14 Mar
  2. 2023Learning without Forgetting for Vision-Language Models30 May
  3. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2016Learning without Forgettingcited by 2,465
  11. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  12. 2013Describing Textures in the Wildcited by 2,110
21 more
  1. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  5. 2021Learning to Prompt for Continual Learningcited by 938
  6. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  7. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  8. 2019Large Scale Incremental Learningcited by 727
  9. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  10. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  11. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  12. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  13. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  14. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  15. 2018Learning without Memorizingcited by 249
  16. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  17. 2020Semantic Drift Compensation for Class-Incremental Learningcited by 241
  18. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 228
  19. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
  20. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 124
  21. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 4; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  2. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  3. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  4. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  5. 2024ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Modelcited by 9
  6. 2024LW2G: Learning Whether to Grow for Prompt-based Continual Learningcited by 9
  7. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  8. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  9. 2024Boosting Open-Domain Continual Learning via Leveraging Intra-domain Category-aware Prototypecited by 4
  10. 2024Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learningcited by 4
  11. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusioncited by 3
  12. 2025Text Classification Under Class Distribution Shift: A Surveycited by 2
3 more
  1. 2026Beyond Classification: Dynamic Adapter Routing for Continual Multimodal Retrievalcited by 1
  2. 2026Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weightingcited by 0
  3. 2026TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MLLM-CL: Continual Learning for Multimodal Large Language Models
  2. 2025Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models
  3. 2024Caption, Create, Continue: Continual Learning with Pre-trained Generative Vision-Language Models
  4. 2025Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models
  5. 2026Black-Box Continual Learning for Vision-Language Models
  6. 2025DesCLIP: Robust Continual Learning via General Attribute Descriptions for VLM-Based Visual Recognition
  7. 2024Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models
  8. 2026Can Vision-Language-Action Models Learn from Real-World Data Continually without Forgetting?