# CLIPCleaner: Cleaning Noisy Labels with CLIP

arXiv: https://arxiv.org/abs/2408.10012 · HTML: https://mlsanity.com/paper/2408.10012/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 72 references that are in the map.
- First posted on arXiv 19 Aug 2024.

## Builds on (most cited first, 30 of 45)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [Decoupling "when to update" from "how to update"](https://mlsanity.com/paper/1706.02613/) (2017)
- [Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise](https://mlsanity.com/paper/1802.05300/) (2018)
- [Probabilistic End-to-end Noise Correction for Learning with Noisy Labels](https://mlsanity.com/paper/1903.07788/) (2019)
- [Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning](https://mlsanity.com/paper/2006.07805/) (2020)
- [Learning with Feature-Dependent Label Noise: A Progressive Approach](https://mlsanity.com/paper/2103.07756/) (2021)
- [Combating Label Noise in Deep Learning Using Abstention](https://mlsanity.com/paper/1905.10964/) (2019)
- [Multi-Objective Interpolation Training for Robustness to Label Noise](https://mlsanity.com/paper/2012.04462/) (2020)
- [Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise](https://mlsanity.com/paper/2012.05458/) (2020)
- [FINE Samples for Learning with Noisy Labels](https://mlsanity.com/paper/2102.11628/) (2021)
- [UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning](https://mlsanity.com/paper/2203.14542/) (2022)
- [A Topological Filter for Learning with Label Noise](https://mlsanity.com/paper/2012.04835/) (2020)

## Built on by (most cited first, 16 of 16)

- [Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization](https://mlsanity.com/paper/2505.00812/) (2025)
- [Sanitizing Manufacturing Dataset Labels Using Vision-Language Models](https://mlsanity.com/paper/2506.23465/) (2025)
- [Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models](https://mlsanity.com/paper/2602.19926/) (2026)
- [Understanding Data Influence with Differential Approximation](https://mlsanity.com/paper/2508.14648/) (2025)
- [Attribution-Guided Model Rectification of Unreliable Neural Network Behaviors](https://mlsanity.com/paper/2603.15656/) (2026)
- [MedTri: A Platform for Structured Medical Report Normalization to Enhance Vision-Language Pretraining](https://mlsanity.com/paper/2602.22143/) (2026)
- [Benchmarking noisy label detection methods](https://mlsanity.com/paper/2510.16211/) (2025)
- [DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models](https://mlsanity.com/paper/2602.19945/) (2026)
- [Deconstructing the Failure of Ideal Noise Correction: A Three-Pillar Diagnosis](https://mlsanity.com/paper/2603.12997/) (2026)
- [Debiased Orthogonal Boundary-Driven Efficient Noise Mitigation](https://mlsanity.com/paper/2410.01944/) (2024)
- [Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection](https://mlsanity.com/paper/2506.14473/) (2025)
- [Weed Out, Then Harvest: Dual Low-Rank Adaptation is an Effective Noisy Label Detector for Noise-Robust Learning](https://mlsanity.com/paper/2510.10208/) (2025)
- [Estimation of Fireproof Structure Class and Construction Year for Disaster Risk Assessment](https://mlsanity.com/paper/2510.22683/) (2025)
- [ACD-U: Asymmetric co-teaching with machine unlearning for robust learning with noisy labels](https://mlsanity.com/paper/2603.07166/) (2026)
- [Noise-Aware Framework for Correcting Corrupted Labels](https://mlsanity.com/paper/2606.11695/) (2026)
- [XMix: Combating Extremely Noisy Labels via Local Smoothness in Self-Supervised Feature Space](https://mlsanity.com/paper/2607.23865/) (2026)

## Abstract (arXiv)

Learning with Noisy labels (LNL) poses a significant challenge for the Machine Learning community. Some of the most widely used approaches that select as clean samples for which the model itself (the in-training model) has high confidence, e.g., `small loss', can suffer from the so called `self-confirmation' bias. This bias arises because the in-training model, is at least partially trained on the noisy labels. Furthermore, in the classification case, an additional challenge arises because some of the label noise is between classes that are visually very similar (`hard noise'). This paper addresses these challenges by proposing a method (textitCLIPCleaner) that leverages CLIP, a powerful Vision-Language (VL) model for constructing a zero-shot classifier for efficient, offline, clean sample selection. This has the advantage that the sample selection is decoupled from the in-training model and that the sample selection is aware of the semantic and visual similarities between the classes due to the way that CLIP is trained. We provide theoretical justifications and empirical evidence to demonstrate the advantages of CLIP for LNL compared to conventional pre-trained models. Compared to current methods that combine iterative sample selection with various techniques, textitCLIPCleaner offers a simple, single-step approach that achieves competitive or superior performance on benchmark datasets. To the best of our knowledge, this is the first time a VL model has been used for sample selection to address the problem of Learning with Noisy Labels (LNL), highlighting their potential in the domain.