# Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?

arXiv: https://arxiv.org/abs/2307.11978 · HTML: https://mlsanity.com/paper/2307.11978/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 55 references that are in the map.
- First posted on arXiv 22 Jul 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [Prompt Distribution Learning](https://mlsanity.com/paper/2205.03340/) (2022)

## Built on by (most cited first, 21 of 21)

- [Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?](https://mlsanity.com/paper/2401.12902/) (2024)
- [Vision-Language Models are Strong Noisy Label Detectors](https://mlsanity.com/paper/2409.19696/) (2024)
- [Promptception: How Sensitive Are Large Multimodal Models to Prompts?](https://mlsanity.com/paper/2509.03986/) (2025)
- [Prototypical Contrastive Learning-based CLIP Fine-tuning for Object Re-identification](https://mlsanity.com/paper/2310.17218/) (2023)
- [Controllable Prompt Tuning For Balancing Group Distributional Robustness](https://mlsanity.com/paper/2403.02695/) (2024)
- [On the Value of Cross-Modal Misalignment in Multimodal Representation Learning](https://mlsanity.com/paper/2504.10143/) (2025)
- [NLPrompt: Noise-Label Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2412.01256/) (2024)
- [Enhancing Vision-Language Few-Shot Adaptation with Negative Learning](https://mlsanity.com/paper/2403.12964/) (2024)
- [Enhancing Domain Adaptation through Prompt Gradient Alignment](https://mlsanity.com/paper/2406.09353/) (2024)
- [Tuning Vision-Language Models with Candidate Labels by Prompt Alignment](https://mlsanity.com/paper/2407.07638/) (2024)
- [Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions](https://mlsanity.com/paper/2506.11111/) (2025)
- [Bayesian Principles Improve Prompt Learning In Vision-Language Models](https://mlsanity.com/paper/2504.14123/) (2025)
- [Understanding the Vulnerability of CLIP to Image Compression](https://mlsanity.com/paper/2311.14029/) (2023)
- [TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models](https://mlsanity.com/paper/2410.05239/) (2024)
- [Curriculum Fine-tuning of Vision Foundation Model for Medical Image Classification Under Label Noise](https://mlsanity.com/paper/2412.00150/) (2024)
- [Efficient Vocabulary-Free Fine-Grained Visual Recognition in the Age of Multimodal LLMs](https://mlsanity.com/paper/2505.01064/) (2025)
- [Exploring Weak-to-Strong Generalization for CLIP-based Classification](https://mlsanity.com/paper/2511.18396/) (2025)
- [Noise-aware few-shot learning through bi-directional multi-view prompt alignment](https://mlsanity.com/paper/2603.11617/) (2026)
- [Holistic Optimal Label Selection for Robust Prompt Learning under Partial Labels](https://mlsanity.com/paper/2604.06614/) (2026)
- [Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision-Language Models](https://mlsanity.com/paper/2605.00591/) (2026)
- [TecoPrompt: Temporal-Conservative Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2609.16858/) (2026)

## Abstract (arXiv)

Vision-language models such as CLIP learn a generic text-image embedding from large-scale training data. A vision-language model can be adapted to a new classification task through few-shot prompt tuning. We find that such a prompt tuning process is highly robust to label noises. This intrigues us to study the key reasons contributing to the robustness of the prompt tuning paradigm. We conducted extensive experiments to explore this property and find the key factors are: 1) the fixed classname tokens provide a strong regularization to the optimization of the model, reducing gradients induced by the noisy samples; 2) the powerful pre-trained image-text embedding that is learned from diverse and generic web data provides strong prior knowledge for image classification. Further, we demonstrate that noisy zero-shot predictions from CLIP can be used to tune its own prompt, significantly enhancing prediction accuracy in the unsupervised setting. The code is available at https://github.com/CEWu/PTNL.