mlsanityData to 7 Oct 2026

Optimizing Active Learning in Vision-Language Models via Parameter-Efficient Uncertainty Calibration

arXiv 2507.21521 ↗First posted 29 Jul 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Active Learning (AL) has emerged as a powerful approach for minimizing labeling costs by selectively sampling the most informative data for neural network model development. Effective AL for large-scale vision-language models necessitates addressing challenges in uncertainty estimation and efficient sampling given the vast number of parameters involved. In this work, we introduce a novel parameter-efficient learning methodology that incorporates uncertainty calibration loss within the AL framework. We propose a differentiable loss function that promotes uncertainty calibration for effectively selecting fewer and most informative data samples for fine-tuning. Through extensive experiments across several datasets and vision backbones, we demonstrate that our solution can match and exceed the performance of complex feature-based sampling techniques while being computationally very efficient. Additionally, we investigate the efficacy of Prompt learning versus Low-rank adaptation (LoRA) in sample selection, providing a detailed comparative analysis of these methods in the context of efficient AL.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 59 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

43 of 59 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Online Zero-Shot Classification with CLIP23 Aug
  2. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models28 May
  3. 2024Revisiting Active Learning in the Era of Vision Foundation Models25 Jan
  4. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models20 Dec
  5. 2023Active Prompt Learning in Vision Language Models18 Nov
  6. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Model7 Nov
  7. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  8. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  10. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  12. 2022MaPLe: Multi-modal Prompt Learningcited by 685
23 more
  1. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  2. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  3. 2020A Survey of Deep Active Learningcited by 547
  4. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  5. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  6. 2019Learning Loss for Active Learningcited by 335
  7. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  8. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  9. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 130
  10. 2020Improving model calibration with accuracy versus uncertainty optimizationcited by 123
  11. 2021Soft Calibration Objectives for Neural Networkscited by 89
  12. 2021Active Learning for Deep Object Detection via Probabilistic Modelingcited by 79
  13. 2022Entropy-based Active Learning for Object Detection with Progressive Diversity Constraintcited by 53
  14. 2024Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Modelscited by 51
  15. 2023Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigmcited by 36
  16. 2023Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Modelcited by 15
  17. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysiscited by 14
  18. 2022Active Learning Over Multiple Domains in Natural Language Taskscited by 11
  19. 2023PELA: Learning Parameter-Efficient Models with Low-Rank Approximationcited by 9
  20. 2024Querying Easily Flip-flopped Samples for Deep Active Learningcited by 7
  21. 2023ActiveDC: Distribution Calibration for Active Finetuningcited by 3
  22. 2021Robust Contrastive Active Learning with Feature-guided Query Strategiescited by 2
  23. 2024Parameter-Efficient Active Learning for Foundational modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Active Learning for Vision-Language Models
  2. 2026Explicit Uncertainty Modeling for Active CLIP Adaptation with Dual Prompt Tuning
  3. 2024Post-hoc Probabilistic Vision-Language Models
  4. 2024Active Prompt Learning with Vision-Language Model Priors
  5. 2026Towards Calibrating Prompt Tuning of Vision-Language Models
  6. 2024Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
  7. 2025Active Learning via Vision-Language Model Adaptation with Open Data
  8. 2024What Makes Good Few-shot Examples for Vision-Language Models?