mlsanityData to 7 Oct 2026

C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion

arXiv 2403.14119 ↗First posted 21 Mar 2024cs.CV cs.AI cs.CLcited by 82

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

In deep learning, test-time adaptation has gained attention as a method for model fine-tuning without the need for labeled data. A prime exemplification is the recently proposed test-time prompt tuning for large-scale vision-language models such as CLIP. Unfortunately, these prompts have been mainly developed to improve accuracy, overlooking the importance of calibration, which is a crucial aspect for quantifying prediction uncertainty. However, traditional calibration methods rely on substantial amounts of labeled data, making them impractical for test-time scenarios. To this end, this paper explores calibration during test-time prompt tuning by leveraging the inherent properties of CLIP. Through a series of observations, we find that the prompt choice significantly affects the calibration in CLIP, where the prompts leading to higher text feature dispersion result in better-calibrated predictions. Introducing the Average Text Feature Dispersion (ATFD), we establish its relationship with calibration error and present a novel method, Calibrated Test-time Prompt Tuning (C-TPT), for optimizing prompts during test-time with enhanced calibration. Through extensive experiments on different CLIP architectures and datasets, we show that C-TPT can effectively improve the calibration of test-time prompt tuning without needing labeled data. The code is publicly accessible at https://github.com/hee-suk-yoon/C-TPT.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 62 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

38 of 62 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization2 Nov
  2. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  3. 2023Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models11 Mar
  4. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2013Describing Textures in the Wildcited by 2,110
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2022Visual Prompt Tuningcited by 1,674
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  10. 2019Natural Adversarial Examplescited by 1,401
  11. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  12. 2016Distribution-Free Predictive Inference For Regressioncited by 657
20 more
  1. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  2. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  3. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  4. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
  5. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 304
  6. 2019Pareto Multi-Task Learningcited by 224
  7. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  8. 2020Learning the Pareto Front with Hypernetworkscited by 122
  9. 2021Soft Calibration Objectives for Neural Networkscited by 89
  10. 2021Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformationcited by 67
  11. 2022LATTE: LAnguage Trajectory TransformErcited by 40
  12. 2023A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Modelscited by 39
  13. 2023Towards Unifying Medical Vision-and-Language Pre-training via Soft Promptscited by 35
  14. 2023What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifierscited by 35
  15. 2023ESD: Expected Squared Difference as a Tuning-Free Trainable Calibration Measurecited by 13
  16. 2024Wavelet-Guided Acceleration of Text Inversion in Diffusion-Based Image Editingcited by 12
  17. 2023INTapt: Information-Theoretic Adversarial Prompt Tuning for Enhanced Non-Native Speech Recognitioncited by 3
  18. 2022SMSMix: Sense-Maintained Sentence Mixup for Word Sense Disambiguationcited by 2
  19. 2023Neutral Editing Framework for Diffusion-based Video Editingcited by 2
  20. 2024AdaMER-CTC: Connectionist Temporal Classification with Adaptive Maximum Entropy Regularization for Automatic Speech Recognitioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 39; 2026: 32.

82 papers.

Sort:

By area:

  1. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  2. 2025Centaur: Robust End-to-End Autonomous Driving with Test-Time Trainingcited by 41
  3. 2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Modelscited by 36
  4. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuningcited by 33
  5. 2024Robust Calibration of Large Vision-Language Adapterscited by 28
  6. 2024Historical Test-time Prompt Tuning for Vision Foundation Modelscited by 21
  7. 2025Bayesian Test-Time Adaptation for Vision-Language Modelscited by 21
  8. 2026AdaJEPA: An Adaptive Latent World Modelcited by 16
  9. 2025O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Modelscited by 14
  10. 2025Learning to Adapt Frozen CLIP for Few-Shot Test-Time Domain Adaptationcited by 14
  11. 2026On-the-Fly VLA Adaptation via Test-Time Reinforcement Learningcited by 14
  12. 2024Post-hoc Probabilistic Vision-Language Modelscited by 13
38 more
  1. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  2. 2024Are foundation models for computer vision good conformal predictors?cited by 12
  3. 2025SCAP: Transductive Test-Time Adaptation via Supportive Clique-based Attribute Promptingcited by 11
  4. 2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimizationcited by 10
  5. 2025ViLU: Learning Vision-Language Uncertainties for Failure Predictioncited by 9
  6. 2025Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Groundingcited by 9
  7. 2024Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Modelcited by 8
  8. 2025Noisy Test-Time Adaptation in Vision-Language Modelscited by 8
  9. 2025Trustworthy Few-Shot Transfer of Medical VLMs through Split Conformal Predictioncited by 8
  10. 2025Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EMcited by 8
  11. 2025TTRV: Test-Time Reinforcement Learning for Vision Language Modelscited by 8
  12. 2025LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Modelscited by 7
  13. 2025Bidirectional Prototype-Reward co-Evolution for Test-Time Adaptation of Vision-Language Modelscited by 6
  14. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  15. 2025Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Modelscited by 6
  16. 2024Is Less More? Exploring Token Condensation as Training-free Test-time Adaptationcited by 5
  17. 2025Noise is an Efficient Learner for Zero-Shot Vision-Language Modelscited by 5
  18. 2025Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignmentcited by 5
  19. 2026Test-Time Adaptation via Cache Personalization for Facial Expression Recognition in Videoscited by 5
  20. 2025Negation-Aware Test-Time Adaptation for Vision-Language Modelscited by 4
  21. 2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvementcited by 4
  22. 2024LI-TTA: Language Informed Test-Time Adaptation for Automatic Speech Recognitioncited by 3
  23. 2024BayesAdapter: enhanced uncertainty estimation in CLIP few-shot adaptationcited by 3
  24. 2025ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updatescited by 3
  25. 2025Calibration-Aware Prompt Learning for Medical Vision-Language Modelscited by 3
  26. 2025A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Modelscited by 3
  27. 2025Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Modelscited by 3
  28. 2025Technical report on label-informed logit redistribution for better domain generalization in low-shot classification with foundation modelscited by 2
  29. 2025Mitigating Cache Noise in Test-Time Adaptation for Large Vision-Language Modelscited by 2
  30. 2025Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibrationcited by 2
  31. 2025D-TPT: Dimensional Entropy Maximization for Calibrating Test-Time Prompt Tuning in Vision-Language Modelscited by 2
  32. 2025Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Modelscited by 2
  33. 2026Cross-modal Proxy Evolving for OOD Detection with Vision-Language Modelscited by 2
  34. 2026SoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuningcited by 2
  35. 2026Dual-Modality Anchor-Guided Filtering for Test-time Prompt Tuningcited by 2
  36. 2026Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretrainingcited by 2
  37. 2026What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspectivecited by 2
  38. 2024Retaining and Enhancing Pre-trained Knowledge in Vision-Language Models with Prompt Ensemblingcited by 1

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
  2. 2024A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test-Time Adaptation for Vision-Language Models
  3. 2025MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models
  4. 2025The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
  5. 2026USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning
  6. 2026Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models
  7. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?
  8. 2024In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Model