mlsanityData to 7 Oct 2026

Active Learning for Vision-Language Models

arXiv 2410.22187 ↗First posted 29 Oct 2024cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Pre-trained vision-language models (VLMs) like CLIP have demonstrated impressive zero-shot performance on a wide range of downstream computer vision tasks. However, there still exists a considerable performance gap between these models and a supervised deep model trained on a downstream dataset. To bridge this gap, we propose a novel active learning (AL) framework that enhances the zero-shot classification performance of VLMs by selecting only a few informative samples from the unlabeled data for annotation during training. To achieve this, our approach first calibrates the predicted entropy of VLMs and then utilizes a combination of self-uncertainty and neighbor-aware uncertainty to calculate a reliable uncertainty measure for active sample selection. Our extensive experiments show that the proposed approach outperforms existing AL approaches on several image classification datasets, and significantly enhances the zero-shot performance of VLMs.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 65 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

50 of 65 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Active Prompt Learning in Vision Language Models18 Nov
  2. 2023Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models11 Mar
  3. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  4. 2022Prompt Distribution Learning6 May
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  11. 2022Visual Prompt Tuningcited by 1,674
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
32 more
  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  3. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  4. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  6. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  7. 2020A Survey of Deep Active Learningcited by 547
  8. 2021Image Segmentation Using Text and Image Promptscited by 509
  9. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  10. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  11. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  12. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  13. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  14. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  15. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  16. 2020Contextual Diversity for Active Learningcited by 139
  17. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 121
  18. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  19. 2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
  20. 2020Active Domain Adaptation via Clustering Uncertainty-weighted Embeddingscited by 71
  21. 2023ALIP: Adaptive Language-Image Pre-training with Synthetic Captioncited by 70
  22. 2020Sequential Graph Convolutional Network for Active Learningcited by 69
  23. 2022Active Learning by Feature Mixingcited by 63
  24. 2023Adversarial Prompt Tuning for Vision-Language Modelscited by 42
  25. 2023Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigmcited by 36
  26. 2022Ontology-enhanced Prompt-tuning for Few-shot Learningcited by 24
  27. 2023Entropic Open-set Active Learningcited by 22
  28. 2022Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approachcited by 16
  29. 2023Investigating the Limitation of CLIP Models: The Worst-Performing Categoriescited by 12
  30. 2023Towards Free Data Selection with General-Purpose Modelscited by 10
  31. 2022Active Learning by Query by Committee with Robust Divergencescited by 9
  32. 2023ActiveDC: Distribution Calibration for Active Finetuningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 6.

15 papers.

Sort:

By area:

  1. 2024Post-hoc Probabilistic Vision-Language Modelscited by 13
  2. 2025Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuningcited by 12
  3. 2025Certainty and Uncertainty Guided Active Domain Adaptationcited by 3
  4. 2025OViP: Online Vision-Language Preference Learning for VLM Hallucinationcited by 2
  5. 2025Active Learning via Vision-Language Model Adaptation with Open Datacited by 2
  6. 2025StepAL: Step-aware Active Learning for Cataract Surgical Videoscited by 2
  7. 2025Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimizationcited by 1
  8. 2026Similarity-as-Evidence: Calibrating Overconfident VLMs for Interpretable and Label-Efficient Medical Active Learningcited by 1
  9. 2026Leveraging Vision-Language Models as Weak Annotators in Active Learningcited by 1
  10. 2025PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Modelscited by 0
  11. 2025Combining Discrepancy-Confusion Uncertainty and Calibration Diversity for Active Fine-Grained Image Classificationcited by 0
  12. 2026Explicit Uncertainty Modeling for Active CLIP Adaptation with Dual Prompt Tuningcited by 0
3 more
  1. 2026Conformal Cross-Modal Active Learningcited by 0
  2. 2026A Human-in-the-Loop Framework for Efficient Prompt Selection in Microscopy Vision-Language Modelscited by 0
  3. 2026Probing Association Instability with Track-State Perturbations for Clip-Level Active Learning in Query-Propagation Multi-Object Trackingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing Active Learning in Vision-Language Models via Parameter-Efficient Uncertainty Calibration
  2. 2024Active Prompt Learning with Vision-Language Model Priors
  3. 2024What Makes Good Few-shot Examples for Vision-Language Models?
  4. 2026Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries
  5. 2025No Labels Needed: Zero-Shot Image Classification with Collaborative Self-Learning
  6. 2024Enabling Small Models for Zero-Shot Selection and Reuse through Model Label Learning
  7. 2022VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models
  8. 2024Bridge the Modality and Capability Gaps in Vision-Language Model Selection