# Contrastive Learning Improves Model Robustness Under Label Noise

arXiv: https://arxiv.org/abs/2104.08984 · HTML: https://mlsanity.com/paper/2104.08984/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 30 of 31)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Bilevel Programming for Hyperparameter Optimization and Meta-Learning](https://mlsanity.com/paper/1806.04910/) (2018)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Joint Optimization Framework for Learning with Noisy Labels](https://mlsanity.com/paper/1803.11364/) (2018)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [Learning to Learn from Noisy Labeled Data](https://mlsanity.com/paper/1812.05214/) (2018)
- [Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates](https://mlsanity.com/paper/1910.03231/) (2019)
- [Learning with Symmetric Label Noise: The Importance of Being Unhinged](https://mlsanity.com/paper/1505.07634/) (2015)
- [Learning From Noisy Large-Scale Datasets With Minimal Supervision](https://mlsanity.com/paper/1701.01619/) (2017)
- [Webly Supervised Learning of Convolutional Networks](https://mlsanity.com/paper/1505.01554/) (2015)
- [Masking: A New Perspective of Noisy Supervision](https://mlsanity.com/paper/1805.08193/) (2018)
- [Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels](https://mlsanity.com/paper/2103.13646/) (2021)
- [Making Risk Minimization Tolerant to Label Noise](https://mlsanity.com/paper/1403.3610/) (2014)
- [L_DMI: An Information-theoretic Noise-robust Loss Function](https://mlsanity.com/paper/1909.03388/) (2019)

## Built on by (most cited first, 30 of 36)

- [Selective-Supervised Contrastive Learning with Noisy Labels](https://mlsanity.com/paper/2203.04181/) (2022)
- [Detecting Corrupted Labels Without Training a Model to Predict](https://mlsanity.com/paper/2110.06283/) (2021)
- [Investigating Why Contrastive Learning Benefits Robustness Against Label Noise](https://mlsanity.com/paper/2201.12498/) (2022)
- [Adaptive Supervised PatchNCE Loss for Learning H&E-to-IHC Stain Translation with Inconsistent Groundtruth Image Pairs](https://mlsanity.com/paper/2303.06193/) (2023)
- [ConDA: Contrastive Domain Adaptation for AI-generated Text Detection](https://mlsanity.com/paper/2309.03992/) (2023)
- [Identifiability of Label Noise Transition Matrix](https://mlsanity.com/paper/2202.02016/) (2022)
- [Mitigating Memorization of Noisy Labels via Regularization between Representations](https://mlsanity.com/paper/2110.09022/) (2021)
- [HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actions](https://mlsanity.com/paper/2304.00387/) (2023)
- [Adaptive Sample Selection for Robust Learning under Label Noise](https://mlsanity.com/paper/2106.15292/) (2021)
- [On Learning Contrastive Representations for Learning with Noisy Labels](https://mlsanity.com/paper/2203.01785/) (2022)
- [When Noisy Labels Meet Long Tail Dilemmas: A Representation Calibration Method](https://mlsanity.com/paper/2211.10955/) (2022)
- [Learning from Noisy Labels with Decoupled Meta Label Purifier](https://mlsanity.com/paper/2302.06810/) (2023)
- [Rethinking Weak Supervision in Helping Contrastive Learning](https://mlsanity.com/paper/2306.04160/) (2023)
- [Unveiling Vulnerabilities of Contrastive Recommender Systems to Poisoning Attacks](https://mlsanity.com/paper/2311.18244/) (2023)
- [L2B: Learning to Bootstrap Robust Models for Combating Label Noise](https://mlsanity.com/paper/2202.04291/) (2022)
- [Learning with Noisy labels via Self-supervised Adversarial Noisy Masking](https://mlsanity.com/paper/2302.06805/) (2023)
- [Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining](https://mlsanity.com/paper/2210.04782/) (2022)
- [Label Noise: Ignorance Is Bliss](https://mlsanity.com/paper/2411.00079/) (2024)
- [PLReMix: Combating Noisy Labels with Pseudo-Label Relaxed Contrastive Representation Learning](https://mlsanity.com/paper/2402.17589/) (2024)
- [How Does Heterogeneous Label Noise Impact Generalization in Neural Nets?](https://mlsanity.com/paper/2106.15475/) (2021)
- [Enhancing Noise-Robust Losses for Large-Scale Noisy Data Learning](https://mlsanity.com/paper/2306.05497/) (2023)
- [USIGAN: Unbalanced Self-Information Feature Transport for Weakly Paired Image IHC Virtual Staining](https://mlsanity.com/paper/2507.05843/) (2025)
- [NoisyActions2M: A Multimedia Dataset for Video Understanding from Noisy Labels](https://mlsanity.com/paper/2110.06827/) (2021)
- [Anatomically Constrained Tractography of the Fetal Brain](https://mlsanity.com/paper/2403.02444/) (2024)
- [Robust Classification with Noisy Labels Based on Posterior Maximization](https://mlsanity.com/paper/2504.06805/) (2025)
- [An Action Is Worth Multiple Words: Handling Ambiguity in Action Recognition](https://mlsanity.com/paper/2210.04933/) (2022)
- [RSVP: Customer Intent Detection via Agent Response Contrastive and Generative Pre-Training](https://mlsanity.com/paper/2310.09773/) (2023)
- [Multi-level Supervised Contrastive Learning](https://mlsanity.com/paper/2502.02202/) (2025)
- [Contrastive Representations for Label Noise Require Fine-Tuning](https://mlsanity.com/paper/2108.09154/) (2021)
- [Contrastive Credibility Propagation for Reliable Semi-Supervised Learning](https://mlsanity.com/paper/2211.09929/) (2022)

## Abstract (arXiv)

Deep neural network-based classifiers trained with the categorical cross-entropy (CCE) loss are sensitive to label noise in the training data. One common type of method that can mitigate the impact of label noise can be viewed as supervised robust methods; one can simply replace the CCE loss with a loss that is robust to label noise, or re-weight training samples and down-weight those with higher loss values. Recently, another type of method using semi-supervised learning (SSL) has been proposed, which augments these supervised robust methods to exploit (possibly) noisy samples more effectively. Although supervised robust methods perform well across different data types, they have been shown to be inferior to the SSL methods on image classification tasks under label noise. Therefore, it remains to be seen that whether these supervised robust methods can also perform well if they can utilize the unlabeled samples more effectively. In this paper, we show that by initializing supervised robust methods using representations learned through contrastive learning leads to significantly improved performance under label noise. Surprisingly, even the simplest method (training a classifier with the CCE loss) can outperform the state-of-the-art SSL method by more than 50% under high label noise when initialized with contrastive learning. Our implementation will be publicly available at urlhttps://github.com/arghosh/noisylabelpretrain.