mlsanityData to 14 Oct 2026

CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models

arXiv 2403.19137 ↗First posted 28 Mar 2024cs.CVcited by 25

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Continual learning (CL) aims to help deep neural networks learn new knowledge while retaining what has been learned. Owing to their powerful generalizability, pre-trained vision-language models such as Contrastive Language-Image Pre-training (CLIP) have lately gained traction as practical CL candidates. However, the domain mismatch between the pre-training and the downstream CL tasks often calls for finetuning of the CLIP on the latter. Most existing finetuning methods exhibit deterministic nature. This makes them overlook the many possible interactions across the input modalities and deems them unsafe for high-risk tasks requiring reliable uncertainty estimation. To address these, our work proposes Continual LeArning with Probabilistic finetuning (CLAP) - a probabilistic modeling framework over visual-guided text features per task, thus providing more calibrated CL finetuning. Unlike recent data-hungry anti-forgetting CL techniques, CLAP alleviates forgetting by exploiting the rich pre-trained knowledge of CLIP for weight initialization and distribution regularization of task-specific parameters. Cooperating with the diverse range of existing prompting methods, CLAP can surpass the predominant deterministic finetuning approaches for CL with CLIP. We conclude with out-of-the-box applications of superior uncertainty estimation abilities of CLAP including novel data detection and exemplar selection within the existing CL setups. Our code is available at urlhttps://github.com/srvCodes/clap4clip.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 83 of its 109 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

83 of 109 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior9 Jan
  2. 2024Learning to Prompt with Text Only Supervision for Vision-Language Models4 Jan
  3. 2023Learning without Forgetting for Vision-Language Models30 May
  4. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar
  5. 2023Debiased Fine-Tuning for Vision-language Models by Prompt Regularization29 Jan
  6. 2022Visual Classification via Description from Large Language Models13 Oct
  7. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  8. 2022Prompt Distribution Learning6 May
  9. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  10. 2021VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts4 Dec
  11. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  12. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 71

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2016Learning without Forgettingcited by 2,496
38 more
  1. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  3. 2013Describing Textures in the Wildcited by 2,125
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  7. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  10. 2021Learning to Prompt for Continual Learningcited by 944
  11. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  12. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  13. 2021Robust fine-tuning of zero-shot modelscited by 797
  14. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  15. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  16. 2019Large Scale Incremental Learningcited by 732
  17. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  18. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  19. 2018End-to-End Incremental Learningcited by 597
  20. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  21. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 449
  22. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  23. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  24. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  25. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  26. 2018Neural Processescited by 311
  27. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  28. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  29. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  30. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 184
  31. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  32. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 164
  33. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
  34. 2018Towards Robust Evaluations of Continual Learningcited by 148
  35. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 141
  36. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 105
  37. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  38. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 88

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 13; 2026: 7.

25 papers.

Sort:

By area:

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  2. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  3. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 18
  4. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  5. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 9
  6. 2024Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learningcited by 6
  7. 2025Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchorscited by 6
  8. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 5
  9. 2025Task-Core Memory Management and Consolidation for Long-term Continual Learningcited by 4
  10. 2025Affordance-First Decomposition for Continual Learning in Video-Language Understandingcited by 4
  11. 2024Mining Your Own Secrets: Diffusion Classifier Scores for Continual Personalization of Text-to-Image Diffusion Modelscited by 3
  12. 2025Toward a Holistic Approach to Continual Model Mergingcited by 3
13 more
  1. 2025Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learningcited by 3
  2. 2025In Praise of Stubbornness: An Empirical Case for Cognitive-Dissonance Aware Continual Update of Knowledge in LLMscited by 2
  3. 2025A Good Start Matters: Enhancing Continual Learning with Data-Driven Weight Initializationcited by 1
  4. 2025Knowledge Graph Enhanced Generative Multi-modal Models for Class-Incremental Learningcited by 1
  5. 2025Understanding Catastrophic Interference: On the Identifibility of Latent Representationscited by 1
  6. 2026Breaking the Limits of Open-Weight CLIP: An Optimization Framework for Self-supervised Fine-tuning of CLIPcited by 1
  7. 2026Continual Learning with Vision-Language Models via Semantic-Geometry Preservationcited by 1
  8. 2026Beyond Classification: Dynamic Adapter Routing for Continual Multimodal Retrievalcited by 1
  9. 2024Less is More: Efficient Model Merging with Binary Task Switchcited by 0
  10. 2026GR4CIL: Gap-compensated Routing for CLIP-based Class Incremental Learningcited by 0
  11. 2026DIMoE-Adapters: Dynamic Expert Evolution for Continual Learning in Vision-Language Modelscited by 0
  12. 2026Listen, Look, and Learn: Learning Without Forgetting through SAM-Audiocited by 0
  13. 2026Curvature-Guided Mixing for MLLM Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mind the Gap: Preserving and Compensating for the Modality Gap in CLIP-Based Continual Learning
  2. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters
  3. 2026MERGETUNE: Continued Fine-Tuning of Vision-Language Models
  4. 2024Caption, Create, Continue: Continual Learning with Pre-trained Generative Vision-Language Models
  5. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusion
  6. 2025Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learning
  7. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Models
  8. 2023CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation