mlsanityData to 7 Oct 2026

Vision-Language Models are Strong Noisy Label Detectors

arXiv 2409.19696 ↗First posted 29 Sep 2024cs.LG cs.CVcited by 16

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent research on fine-tuning vision-language models has demonstrated impressive performance in various downstream tasks. However, the challenge of obtaining accurately labeled data in real-world applications poses a significant obstacle during the fine-tuning process. To address this challenge, this paper presents a Denoising Fine-Tuning framework, called DeFT, for adapting vision-language models. DeFT utilizes the robust alignment of textual and visual features pre-trained on millions of auxiliary image-text pairs to sieve out noisy labels. The proposed framework establishes a noisy label detector by learning positive and negative textual prompts for each class. The positive prompt seeks to reveal distinctive features of the class, while the negative prompt serves as a learnable threshold for separating clean and noisy samples. We employ parameter-efficient fine-tuning for the adaptation of a pre-trained visual encoder to promote its alignment with the learned textual prompts. As a general framework, DeFT can seamlessly fine-tune many pre-trained models to downstream tasks by utilizing carefully selected clean samples. Experimental results on seven synthetic and real-world noisy datasets validate the effectiveness of DeFT in both noisy label detection and image classification.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 51 references that are in the map. First posted on arXiv 29 Sep 2024.

Builds on

41 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?22 Jul
  2. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  3. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2022Visual Prompt Tuningcited by 1,674
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  11. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  12. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
26 more
  1. 2017A Closer Look at Memorization in Deep Networkscited by 936
  2. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  3. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 691
  4. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  5. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  6. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
  7. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
  8. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 414
  9. 2020Combating noisy labels by agreement: A joint training method with co-regularizationcited by 289
  10. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
  11. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 253
  12. 2019NLNL: Negative Learning for Noisy Labelscited by 151
  13. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  14. 2020A Survey of Label-noise Representation Learning: Past, Present and Futurecited by 121
  15. 2022DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotationscited by 102
  16. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  17. 2021Sample Selection with Uncertainty of Losses for Learning with Noisy Labelscited by 79
  18. 2022UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learningcited by 77
  19. 2021NGC: A Unified Framework for Learning with Open-World Noisy Datacited by 58
  20. 2022Learning with Neighbor Consistency for Noisy Labelscited by 55
  21. 2022ProMix: Combating Label Noise via Maximizing Clean Sample Utilitycited by 48
  22. 2021LongReMix: Robust Learning with High Confidence Samples in a Noisy Label Environmentcited by 40
  23. 2021Robust Long-Tailed Learning under Label Noisecited by 35
  24. 2023Visual Tuningcited by 34
  25. 2023DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition with Limited Annotationscited by 22
  26. 2023Fine tuning Pre trained Models for Robustness Under Noisy Labelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 5.

16 papers.

Sort:

By area:

  1. 2024A CLIP-Powered Framework for Robust and Generalizable Data Selectioncited by 20
  2. 2025LADA: Scalable Label-Specific CLIP Adapter for Continual Learningcited by 11
  3. 2025Preserving AUC Fairness in Learning with Noisy Protected Groupscited by 6
  4. 2025Sanitizing Manufacturing Dataset Labels Using Vision-Language Modelscited by 6
  5. 2025Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimizationcited by 5
  6. 2025When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identificationcited by 5
  7. 2025Weakly-Supervised Contrastive Learning for Imprecise Class Labelscited by 1
  8. 2025Benchmarking noisy label detection methodscited by 1
  9. 2026Nonparametric Teaching of Attention Learnerscited by 1
  10. 2024Debiased Orthogonal Boundary-Driven Efficient Noise Mitigationcited by 0
  11. 2025Is Supervised Learning Really That Different from Unsupervised?cited by 0
  12. 2025Dual-Path Stable Soft Prompt Generation for Domain Generalizationcited by 0
4 more
  1. 2026ACD-U: Asymmetric co-teaching with machine unlearning for robust learning with noisy labelscited by 0
  2. 2026Noise-aware few-shot learning through bi-directional multi-view prompt alignmentcited by 0
  3. 2026SHED: Style-Homogenized Embedding Alignment for Domain Generalizationcited by 0
  4. 2026On Revisiting Entropy for Identifying Mislabeled Imagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Fine-tuning Pre-trained Vision-Language Models in a Human-Annotation-Free Manner
  2. 2026Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision-Language Models
  3. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  4. 2023Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning
  5. 2024Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
  6. 2024NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
  7. 2024Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data
  8. 2025Noise is an Efficient Learner for Zero-Shot Vision-Language Models