mlsanityData to 7 Oct 2026

Towards Calibrating Prompt Tuning of Vision-Language Models

arXiv 2602.19024 ↗First posted 22 Feb 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Prompt tuning of large-scale vision-language models such as CLIP enables efficient task adaptation without updating model weights. However, it often leads to poor confidence calibration and unreliable predictive uncertainty. We address this problem by proposing a calibration framework that enhances predictive reliability while preserving the geometry of the pretrained CLIP embedding space, which is required for robust generalization. Our approach extends the standard cross-entropy loss with two complementary regularizers: (1) a mean-variance margin penalty that stabilizes inter-class logit margins by maximizing their average while minimizing dispersion, mitigating underconfidence and overconfidence spikes; and (2) a text moment-matching loss that aligns the first and second moments of tuned text embeddings with their frozen CLIP counterparts, preserving semantic dispersion crucial for generalization. Through extensive experiments across 7 prompt-tuning methods and 11 diverse datasets, we demonstrate that our approach significantly reduces the Expected Calibration Error (ECE) compared to competitive calibration techniques on both base and novel classes

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 47 references that are in the map. First posted on arXiv 22 Feb 2026.

Builds on

28 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion21 Mar
  2. 2024Open-Vocabulary Calibration for Fine-tuned CLIP7 Feb
  3. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  4. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2013Describing Textures in the Wildcited by 2,110
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  6. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2019Natural Adversarial Examplescited by 1,401
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  10. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  11. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  12. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
10 more
  1. 2019Measuring Calibration in Deep Learningcited by 443
  2. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  3. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  4. 2021The Devil is in the Margin: Margin-based Label Smoothing for Network Calibrationcited by 91
  5. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 72
  6. 2022Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysiscited by 62
  7. 2023CLIP in Medical Imaging: A Surveycited by 40
  8. 2024Robust Calibration of Large Vision-Language Adapterscited by 28
  9. 2025Vision-Language Models for Autonomous Driving: CLIP-Based Dynamic Scene Understandingcited by 9
  10. 2024Uncertainty-Aware Prediction and Application in Planning for Autonomous Driving: Definitions, Methods, and Comparisoncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
  2. 2026Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining
  3. 2024Generalizable Prompt Tuning for Vision-Language Models
  4. 2025Prompt Tuning Vision Language Models with Margin Regularizer for Few-Shot Learning under Distribution Shifts
  5. 2023Distribution-Aware Prompt Tuning for Vision-Language Models
  6. 2025Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment
  7. 2026ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models
  8. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models