mlsanityData to 7 Oct 2026

An Empirical Study Into What Matters for Calibrating Vision-Language Models

arXiv 2402.07417 ↗First posted 12 Feb 2024cs.CV cs.LGcited by 20

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-Language Models (VLMs) have emerged as the dominant approach for zero-shot recognition, adept at handling diverse scenarios and significant distribution changes. However, their deployment in risk-sensitive areas requires a deeper understanding of their uncertainty estimation capabilities, a relatively uncharted area. In this study, we explore the calibration properties of VLMs across different architectures, datasets, and training strategies. In particular, we analyze the uncertainty estimation performance of VLMs when calibrated in one domain, label set or hierarchy level, and tested in a different one. Our findings reveal that while VLMs are not inherently calibrated for uncertainty, temperature scaling significantly and consistently improves calibration, even across shifts in distribution and changes in label set. Moreover, VLMs can be calibrated with a very small set of examples. Through detailed experimentation, we highlight the potential applications and importance of our insights, aiming for more reliable and effective use of VLMs in critical, real-world scenarios.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 75 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

54 of 75 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023ChatGPT-Powered Hierarchical Comparisons for Image Classification1 Nov
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2017On Calibration of Modern Neural Networkscited by 4,860
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2022A ConvNet for the 2020scited by 3,545
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  13. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  14. 2021Robust fine-tuning of zero-shot modelscited by 785
  15. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  16. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  17. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  18. 2021ImageNet-21K Pretraining for the Massescited by 607
  19. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  20. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  21. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  22. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  23. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  24. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  25. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  26. 2023Demystifying CLIP Datacited by 265
  27. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  28. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 197
  29. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  30. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  31. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  32. 2020Calibration of Neural Networks using Splinescited by 97
  33. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  34. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  35. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  36. 2015Posterior calibration and exploratory analysis for natural language processing modelscited by 75
  37. 2019Do Image Classifiers Generalize Across Time?cited by 59
  38. 2022Robust Calibration with Multi-domain Temperature Scalingcited by 39

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 8; 2026: 5.

20 papers.

Sort:

By area:

  1. 2024DOTA: Distributional Test-Time Adaptation of Vision-Language Modelscited by 22
  2. 2024Self-Calibrated Tuning of Vision-Language Models for Out-of-Distribution Detectioncited by 15
  3. 2025Conformal Prediction for Zero-Shot Modelscited by 14
  4. 2024Post-hoc Probabilistic Vision-Language Modelscited by 13
  5. 2024Are foundation models for computer vision good conformal predictors?cited by 12
  6. 2025Trustworthy Few-Shot Transfer of Medical VLMs through Split Conformal Predictioncited by 8
  7. 2025Noise is an Efficient Learner for Zero-Shot Vision-Language Modelscited by 5
  8. 2026Overconfidence and Calibration in Medical VQA: Empirical Findings and Hallucination-Aware Mitigationcited by 5
  9. 2024VORD: Visual Ordinal Calibration for Mitigating Object Hallucinations in Large Vision-Language Modelscited by 4
  10. 2025FLOSS: Free Lunch in Open-vocabulary Semantic Segmentationcited by 4
  11. 2024BayesAdapter: enhanced uncertainty estimation in CLIP few-shot adaptationcited by 3
  12. 2026Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Modelscited by 3
8 more
  1. 2024FastRM: An efficient and automatic explainability framework for multimodal generative modelscited by 2
  2. 2025ZeST: an VLM-based Zero-Shot Traversability Navigation for Unknown Environmentscited by 2
  3. 2025Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Modelscited by 2
  4. 2025Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Modelscited by 2
  5. 2026SoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuningcited by 2
  6. 2026ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Modelscited by 1
  7. 2025Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspectivecited by 0
  8. 2026Target-Checked Reliability Score Refinement for Video Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Open-Vocabulary Calibration for Fine-tuned CLIP
  2. 2024Uncertainty-Aware Evaluation for Vision-Language Models
  3. 2026Respect Your Zero-Shot Uncertainty: Conservative Calibration for Test-Time-Adapted Vision-Language Models
  4. 2025VSCBench: Bridging the Gap in Vision-Language Model Safety Calibration
  5. 2023Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity
  6. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
  7. 2026Validating, Not Sampling: Region-Level Robustness of Vision-Language and Vision-Language-Action Models
  8. 2025Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy