mlsanityData to 7 Oct 2026

Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models

arXiv 2303.12748 ↗First posted 11 Mar 2023cs.CV cs.LGcited by 17

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Calibration of deep learning models is crucial to their trustworthiness and safe usage, and as such, has been extensively studied in supervised classification models, with methods crafted to decrease miscalibration. However, there has yet to be a comprehensive study of the calibration of vision-language models that are used for zero-shot inference, like CLIP. We measure calibration across relevant variables like prompt, dataset, and architecture, and find that zero-shot inference with CLIP is miscalibrated. Furthermore, we propose a modified version of temperature scaling that is aligned with the common use cases of CLIP as a zero-shot inference model, and show that a single learned temperature generalizes for each specific CLIP model (defined by a chosen pre-training dataset and architecture) across inference dataset and prompt choice.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 21 references that are in the map. First posted on arXiv 11 Mar 2023.

Builds on

10 of 21 references.

References · 10

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  7. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  8. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  9. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  10. 2019Accurate Layerwise Interpretable Competence Estimationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 4; 2025: 7; 2026: 2.

17 papers.

Sort:

By area:

  1. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersioncited by 82
  2. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 72
  3. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  4. 2023Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuningcited by 30
  5. 2023A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shiftcited by 17
  6. 2024Open-Vocabulary Calibration for Fine-tuned CLIPcited by 15
  7. 2024Active Learning for Vision-Language Modelscited by 15
  8. 2024Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Modelscited by 9
  9. 2025Semantic Consistent Language Gaussian Splatting for Point-Level Open-vocabulary Queryingcited by 4
  10. 2026RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinementcited by 4
  11. 2025Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibrationcited by 2
  12. 2025ZeST: an VLM-based Zero-Shot Traversability Navigation for Unknown Environmentscited by 2
5 more
  1. 2025Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Modelscited by 2
  2. 2025Calibrating Uncertainty for Zero-Shot Adversarial CLIPcited by 1
  3. 2026Similarity-as-Evidence: Calibrating Overconfident VLMs for Interpretable and Label-Efficient Medical Active Learningcited by 1
  4. 2025SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptationcited by 0
  5. 2025Enabling Fine-Grained Operating Points for Black-Box LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Respect Your Zero-Shot Uncertainty: Conservative Calibration for Test-Time-Adapted Vision-Language Models
  2. 2026Towards Calibrating Prompt Tuning of Vision-Language Models
  3. 2024An Empirical Study Into What Matters for Calibrating Vision-Language Models
  4. 2023Robust Fine-Tuning of Vision-Language Models for Domain Generalization
  5. 2026Domain Recentering and Confidence-Weighted Prior Calibration for Vision-Language Models
  6. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting
  7. 2022VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models
  8. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?