mlsanityData to 14 Oct 2026

Contrastive Learning Improves Model Robustness Under Label Noise

arXiv 2104.08984 ↗First posted 19 Apr 2021cs.CV cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Deep neural network-based classifiers trained with the categorical cross-entropy (CCE) loss are sensitive to label noise in the training data. One common type of method that can mitigate the impact of label noise can be viewed as supervised robust methods; one can simply replace the CCE loss with a loss that is robust to label noise, or re-weight training samples and down-weight those with higher loss values. Recently, another type of method using semi-supervised learning (SSL) has been proposed, which augments these supervised robust methods to exploit (possibly) noisy samples more effectively. Although supervised robust methods perform well across different data types, they have been shown to be inferior to the SSL methods on image classification tasks under label noise. Therefore, it remains to be seen that whether these supervised robust methods can also perform well if they can utilize the unlabeled samples more effectively. In this paper, we show that by initializing supervised robust methods using representations learned through contrastive learning leads to significantly improved performance under label noise. Surprisingly, even the simplest method (training a classifier with the CCE loss) can outperform the state-of-the-art SSL method by more than 50% under high label noise when initialized with contrastive learning. Our implementation will be publicly available at urlhttps://github.com/arghosh/noisylabelpretrain.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 19 Apr 2021.

Builds on

31 of 40 references.

References · 31

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  3. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  5. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  6. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  7. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 692
  8. 2018Learning to Reweight Examples for Robust Deep Learningcited by 675
  9. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 665
  10. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 653
  11. 2017Large Batch Training of Convolutional Networkscited by 535
  12. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
19 more
  1. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 443
  2. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 416
  3. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  4. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  5. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  6. 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 328
  7. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  8. 2018Joint Optimization Framework for Learning with Noisy Labelscited by 293
  9. 2020Normalized Loss Functions for Deep Learning with Noisy Labelscited by 257
  10. 2018Learning to Learn from Noisy Labeled Datacited by 160
  11. 2019Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Ratescited by 142
  12. 2015Learning with Symmetric Label Noise: The Importance of Being Unhingedcited by 120
  13. 2017Learning From Noisy Large-Scale Datasets With Minimal Supervisioncited by 115
  14. 2015Webly Supervised Learning of Convolutional Networkscited by 95
  15. 2018Masking: A New Perspective of Noisy Supervisioncited by 93
  16. 2021Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labelscited by 69
  17. 2014Making Risk Minimization Tolerant to Label Noisecited by 54
  18. 2019L_DMI: An Information-theoretic Noise-robust Loss Functioncited by 26
  19. 2013Double Ramp Loss Based Reject Option Classifiercited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 9; 2023: 10; 2024: 4; 2025: 5; 2026: 2.

36 papers.

Sort:

By area:

  1. 2022Selective-Supervised Contrastive Learning with Noisy Labelscited by 112
  2. 2021Detecting Corrupted Labels Without Training a Model to Predictcited by 58
  3. 2022Investigating Why Contrastive Learning Benefits Robustness Against Label Noisecited by 46
  4. 2023Adaptive Supervised PatchNCE Loss for Learning H&E-to-IHC Stain Translation with Inconsistent Groundtruth Image Pairscited by 38
  5. 2023ConDA: Contrastive Domain Adaptation for AI-generated Text Detectioncited by 37
  6. 2022Identifiability of Label Noise Transition Matrixcited by 34
  7. 2021Mitigating Memorization of Noisy Labels via Regularization between Representationscited by 26
  8. 2023HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actionscited by 25
  9. 2021Adaptive Sample Selection for Robust Learning under Label Noisecited by 21
  10. 2022On Learning Contrastive Representations for Learning with Noisy Labelscited by 21
  11. 2022When Noisy Labels Meet Long Tail Dilemmas: A Representation Calibration Methodcited by 21
  12. 2023Learning from Noisy Labels with Decoupled Meta Label Purifiercited by 21
24 more
  1. 2023Rethinking Weak Supervision in Helping Contrastive Learningcited by 18
  2. 2023Unveiling Vulnerabilities of Contrastive Recommender Systems to Poisoning Attackscited by 11
  3. 2022L2B: Learning to Bootstrap Robust Models for Combating Label Noisecited by 10
  4. 2023Learning with Noisy labels via Self-supervised Adversarial Noisy Maskingcited by 10
  5. 2022Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretrainingcited by 9
  6. 2024Label Noise: Ignorance Is Blisscited by 9
  7. 2024PLReMix: Combating Noisy Labels with Pseudo-Label Relaxed Contrastive Representation Learningcited by 5
  8. 2021How Does Heterogeneous Label Noise Impact Generalization in Neural Nets?cited by 3
  9. 2023Enhancing Noise-Robust Losses for Large-Scale Noisy Data Learningcited by 3
  10. 2025USIGAN: Unbalanced Self-Information Feature Transport for Weakly Paired Image IHC Virtual Stainingcited by 3
  11. 2021NoisyActions2M: A Multimedia Dataset for Video Understanding from Noisy Labelscited by 2
  12. 2024Anatomically Constrained Tractography of the Fetal Braincited by 2
  13. 2025Robust Classification with Noisy Labels Based on Posterior Maximizationcited by 2
  14. 2022An Action Is Worth Multiple Words: Handling Ambiguity in Action Recognitioncited by 1
  15. 2023RSVP: Customer Intent Detection via Agent Response Contrastive and Generative Pre-Trainingcited by 1
  16. 2025Multi-level Supervised Contrastive Learningcited by 1
  17. 2021Contrastive Representations for Label Noise Require Fine-Tuningcited by 0
  18. 2022Contrastive Credibility Propagation for Reliable Semi-Supervised Learningcited by 0
  19. 2023ViewMix: Augmentation for Robust Representation in Self-Supervised Learningcited by 0
  20. 2024Addressing Long-Tail Noisy Label Learning Problems: a Two-Stage Solution with Label Refurbishment Considering Label Raritycited by 0
  21. 2025Set a Thief to Catch a Thief: Combating Label Noise through Noisy Meta Learningcited by 0
  22. 2025Leveraging Adversarial Learning for Pathological Fidelity in Virtual Stainingcited by 0
  23. 2026Mitigating Instance Entanglement in Instance-Dependent Partial Label Learningcited by 0
  24. 2026Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalancecited by 0