mlsanityData to 7 Oct 2026

Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?

arXiv 2307.11978 ↗First posted 22 Jul 2023cs.CV cs.AI cs.LGcited by 21

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models such as CLIP learn a generic text-image embedding from large-scale training data. A vision-language model can be adapted to a new classification task through few-shot prompt tuning. We find that such a prompt tuning process is highly robust to label noises. This intrigues us to study the key reasons contributing to the robustness of the prompt tuning paradigm. We conducted extensive experiments to explore this property and find the key factors are: 1) the fixed classname tokens provide a strong regularization to the optimization of the model, reducing gradients induced by the noisy samples; 2) the powerful pre-trained image-text embedding that is learned from diverse and generic web data provides strong prior knowledge for image classification. Further, we demonstrate that noisy zero-shot predictions from CLIP can be used to tune its own prompt, significantly enhancing prediction accuracy in the unsupervised setting. The code is available at https://github.com/CEWu/PTNL.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 55 references that are in the map. First posted on arXiv 22 Jul 2023.

Builds on

41 of 55 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  2. 2022Prompt Distribution Learning6 May
  3. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  9. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  12. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
24 more
  1. 2021Learning to Prompt for Continual Learningcited by 938
  2. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  3. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  4. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  5. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  6. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
  7. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  8. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
  9. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  10. 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 327
  11. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  12. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  13. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  14. 2020Normalized Loss Functions for Deep Learning with Noisy Labelscited by 257
  15. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  16. 2019Probabilistic End-to-end Noise Correction for Learning with Noisy Labelscited by 210
  17. 2018Dimensionality-Driven Learning with Noisy Labelscited by 173
  18. 2017Learning from Noisy Labels with Distillationcited by 160
  19. 2018Learning to Learn from Noisy Labeled Datacited by 160
  20. 2017Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samplescited by 147
  21. 2020Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learningcited by 142
  22. 2019Meta Label Correction for Noisy Label Learningcited by 108
  23. 2022DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotationscited by 102
  24. 2022Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videoscited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 6; 2026: 4.

21 papers.

Sort:

By area:

  1. 2024Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?cited by 31
  2. 2024Vision-Language Models are Strong Noisy Label Detectorscited by 16
  3. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
  4. 2023Prototypical Contrastive Learning-based CLIP Fine-tuning for Object Re-identificationcited by 11
  5. 2024Controllable Prompt Tuning For Balancing Group Distributional Robustnesscited by 10
  6. 2024NLPrompt: Noise-Label Prompt Learning for Vision-Language Modelscited by 9
  7. 2025On the Value of Cross-Modal Misalignment in Multimodal Representation Learningcited by 9
  8. 2024Enhancing Vision-Language Few-Shot Adaptation with Negative Learningcited by 8
  9. 2024Enhancing Domain Adaptation through Prompt Gradient Alignmentcited by 8
  10. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignmentcited by 6
  11. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  12. 2023Understanding the Vulnerability of CLIP to Image Compressioncited by 3
9 more
  1. 2024TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Modelscited by 3
  2. 2024Curriculum Fine-tuning of Vision Foundation Model for Medical Image Classification Under Label Noisecited by 3
  3. 2025Bayesian Principles Improve Prompt Learning In Vision-Language Modelscited by 3
  4. 2025Efficient Vocabulary-Free Fine-Grained Visual Recognition in the Age of Multimodal LLMscited by 2
  5. 2025Exploring Weak-to-Strong Generalization for CLIP-based Classificationcited by 0
  6. 2026Noise-aware few-shot learning through bi-directional multi-view prompt alignmentcited by 0
  7. 2026Holistic Optimal Label Selection for Robust Prompt Learning under Partial Labelscited by 0
  8. 2026Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision-Language Modelscited by 0
  9. 2026TecoPrompt: Temporal-Conservative Prompt Learning for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Prompt Tuning Vision Language Models with Margin Regularizer for Few-Shot Learning under Distribution Shifts
  2. 2025Robust Prompt Tuning for Vision-Language Models with Mild Semantic Noise
  3. 2024Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning
  4. 2023Towards Robust Prompts on Vision-Language Models
  5. 2025Debiased Prompt Tuning in Vision-Language Model without Annotations
  6. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  7. 2025Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibration
  8. 2023Distribution-Aware Prompt Tuning for Vision-Language Models