mlsanityData to 7 Oct 2026

Don't Stop Learning: Towards Continual Learning for the CLIP Model

arXiv 2207.09248 ↗First posted 19 Jul 2022cs.CVcited by 61

Not on the map. Why?

Cited by 29 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The Contrastive Language-Image Pre-training (CLIP) Model is a recently proposed large-scale pre-train model which attracts increasing attention in the computer vision community. Benefiting from its gigantic image-text training set, the CLIP model has learned outstanding capabilities in zero-shot learning and image-text matching. To boost the recognition performance of CLIP on some target visual concepts, it is often desirable to further update the CLIP model by fine-tuning some classes-of-interest on extra training data. This operation, however, raises an important concern: will the update hurt the zero-shot learning or image-text matching capability of the CLIP, i.e., the catastrophic forgetting issue? If yes, could existing continual learning algorithms be adapted to alleviate the risk of catastrophic forgetting? To answer these questions, this work conducts a systemic study on the continual learning issue of the CLIP model. We construct evaluation protocols to measure the impact of fine-tuning updates and explore different ways to upgrade existing continual learning methods to mitigate the forgetting issue of the CLIP model. Our study reveals the particular challenges of CLIP continual learning problem and lays a foundation for further researches. Moreover, we propose a new algorithm, dubbed Learning without Forgetting via Replayed Vocabulary (VR-LwF), which shows exact effectiveness for alleviating the forgetting issue of the CLIP model.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 44 of its 57 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

44 of 57 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2022Generative Adversarial Networkscited by 5,603
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  11. 2013Auto-Encoding Variational Bayescited by 3,980
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
32 more
  1. 2016Learning without Forgettingcited by 2,465
  2. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2017Continual Learning with Deep Generative Replaycited by 1,152
  9. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  11. 2021Robust fine-tuning of zero-shot modelscited by 785
  12. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  13. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  14. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  15. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  16. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  17. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  18. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  19. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  20. 2018Learning without Memorizingcited by 249
  21. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  22. 2018Selective Experience Replay for Lifelong Learningcited by 239
  23. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  24. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  25. 2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 220
  26. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  27. 2021On Learning the Geodesic Path for Incremental Learningcited by 85
  28. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  29. 2019Overcoming Catastrophic Forgetting by Neuron-level Plasticity Controlcited by 36
  30. 2018Continual Classification Learning Using Generative Modelscited by 28
  31. 2020Triple Memory Networks: a Brain-Inspired Method for Continual Learningcited by 28
  32. 2021Rectification-based Knowledge Retention for Continual Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 11; 2024: 21; 2025: 15; 2026: 11.

61 papers.

Sort:

By area:

  1. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  2. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  3. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  4. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  5. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 162
  6. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  7. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 93
  8. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  9. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  10. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  11. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  12. 2023AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learningcited by 46
38 more
  1. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  2. 2023TiC-CLIP: Continual Training of CLIP Modelscited by 38
  3. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Modelscited by 34
  4. 2024VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Modelcited by 23
  5. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillationcited by 18
  6. 2024MUNBa: Machine Unlearning via Nash Bargainingcited by 17
  7. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  8. 2024CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learningcited by 15
  9. 2023Controllable Textual Inversion for Personalized Text-to-Image Generationcited by 14
  10. 2024Learning More Generalized Experts by Merging Experts in Mixture-of-Expertscited by 12
  11. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  12. 2025Continual Text-to-Video Retrieval with Frame Fusion and Task-Aware Routingcited by 12
  13. 2024OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Imagescited by 9
  14. 2024Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learningcited by 7
  15. 2025IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Promptingcited by 7
  16. 2024Brain-inspired continual pre-trained learner via silent synaptic consolidationcited by 6
  17. 2025Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchorscited by 6
  18. 2024A streamlined Approach to Multimodal Few-Shot Class Incremental Learning for Fine-Grained Datasetscited by 4
  19. 2024Boosting Open-Domain Continual Learning via Leveraging Intra-domain Category-aware Prototypecited by 4
  20. 2024Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learningcited by 4
  21. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 4
  22. 2024Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Featurescited by 3
  23. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusioncited by 3
  24. 2025Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Modelscited by 3
  25. 2025Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgettingcited by 3
  26. 2024Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Modelcited by 2
  27. 2024Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learningcited by 2
  28. 2025Satellites Reveal Mobility: A Commuting Origin-destination Flow Generator for Global Citiescited by 2
  29. 2025iDPA: Instance Decoupled Prompt Attention for Incremental Medical Object Detectioncited by 2
  30. 2024The Inter-Intra Modal Measure: A Predictive Lens on Fine-Tuning Outcomes in Vision-Language Modelscited by 1
  31. 2024VLMs meet UDA: Boosting Transferability of Open Vocabulary Segmentation with Unsupervised Domain Adaptationcited by 1
  32. 2025Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Modelscited by 1
  33. 2025MedPEFT-CL: Dual-Phase Parameter-Efficient Continual Learning with Medical Semantic Adapter and Bidirectional Memory Consolidationcited by 1
  34. 2026Breaking the Limits of Open-Weight CLIP: An Optimization Framework for Self-supervised Fine-tuning of CLIPcited by 1
  35. 2026Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRAcited by 1
  36. 2026Continual Learning with Vision-Language Models via Semantic-Geometry Preservationcited by 1
  37. 2023Big-model Driven Few-shot Continual Learningcited by 0
  38. 2024Quantified Task Misalignment to Inform PEFT: An Exploration of Domain Generalization and Catastrophic Forgetting in CLIPcited by 0

50 most cited of 61 All 61 (JSON)