# Consistency Regularization Can Improve Robustness to Label Noise

arXiv: https://arxiv.org/abs/2110.01242 · HTML: https://mlsanity.com/paper/2110.01242/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 32 references that are in the map.
- First posted on arXiv 4 Oct 2021.

## Builds on (most cited first, 29 of 29)

- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Joint Optimization Framework for Learning with Noisy Labels](https://mlsanity.com/paper/1803.11364/) (2018)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [Does label smoothing mitigate label noise?](https://mlsanity.com/paper/2003.02819/) (2020)
- [SELF: Learning to Filter Noisy Labels with Self-Ensembling](https://mlsanity.com/paper/1910.01842/) (2019)
- [Learning to Learn from Noisy Labeled Data](https://mlsanity.com/paper/1812.05214/) (2018)
- [Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks](https://mlsanity.com/paper/1903.11680/) (2019)
- [Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates](https://mlsanity.com/paper/1910.03231/) (2019)
- [Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks](https://mlsanity.com/paper/1706.00038/) (2017)
- [Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels](https://mlsanity.com/paper/2105.04522/) (2021)
- [Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels](https://mlsanity.com/paper/2103.13646/) (2021)
- [Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels](https://mlsanity.com/paper/1705.01936/) (2017)
- [When Optimizing f-divergence is Robust with Label Noise](https://mlsanity.com/paper/2011.03687/) (2020)
- [Graph convolutional networks for learning with few clean and many noisy labels](https://mlsanity.com/paper/1910.00324/) (2019)

## Built on by (most cited first, 18 of 18)

- [Neighborhood Collective Estimation for Noisy Label Identification and Correction](https://mlsanity.com/paper/2208.03207/) (2022)
- [Robust Data Pruning under Label Noise via Maximizing Re-labeling Accuracy](https://mlsanity.com/paper/2311.01002/) (2023)
- [CSOT: Curriculum and Structure-Aware Optimal Transport for Learning with Noisy Labels](https://mlsanity.com/paper/2312.06221/) (2023)
- [Temporally Consistent Koopman Autoencoders for Forecasting Dynamical Systems](https://mlsanity.com/paper/2403.12335/) (2024)
- [Elucidating the design space of language models for image generation](https://mlsanity.com/paper/2410.16257/) (2024)
- [Data Upcycling Knowledge Distillation for Image Super-Resolution](https://mlsanity.com/paper/2309.14162/) (2023)
- [Label Noise: Correcting the Forward-Correction](https://mlsanity.com/paper/2307.13100/) (2023)
- [Enhancing the Scalability of Classical Surrogates for Real-World Quantum Machine Learning Applications](https://mlsanity.com/paper/2508.06131/) (2025)
- [Noisy Label Classification using Label Noise Selection with Test-Time Augmentation Cross-Entropy and NoiseMix Learning](https://mlsanity.com/paper/2212.00479/) (2022)
- [Prioritizing Informative Features and Examples for Deep Learning from Noisy Data](https://mlsanity.com/paper/2403.00013/) (2024)
- [Unsupervised Domain Adaptation for Action Recognition via Self-Ensembling and Conditional Embedding Alignment](https://mlsanity.com/paper/2410.17489/) (2024)
- [Beyond Self-Consistency: Loss-Balanced Perturbation-Based Regularization Improves Industrial-Scale Ads Ranking](https://mlsanity.com/paper/2502.18478/) (2025)
- [AVP-Fusion: Adaptive Multi-Modal Fusion and Contrastive Learning for Two-Stage Antiviral Peptide Identification](https://mlsanity.com/paper/2512.21544/) (2025)
- [AVP-Pro: An Adaptive Multi-Modal Fusion and Contrastive Learning Approach for Comprehensive Two-Stage Antiviral Peptide Identification](https://mlsanity.com/paper/2601.11028/) (2026)
- [Easy Samples Are All You Need: Self-Evolving LLMs via Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2604.18639/) (2026)
- [Voice Biomarkers for Depression and Anxiety](https://mlsanity.com/paper/2605.09908/) (2026)
- [RIVET: Robust Idempotent Voice Attribute Editing](https://mlsanity.com/paper/2606.19629/) (2026)
- [Encoder-Decoder Manifold Alignment for Idempotent Generation](https://mlsanity.com/paper/2606.22304/) (2026)

## Abstract (arXiv)

Consistency regularization is a commonly-used technique for semi-supervised and self-supervised learning. It is an auxiliary objective function that encourages the prediction of the network to be similar in the vicinity of the observed training samples. Hendrycks et al. (2020) have recently shown such regularization naturally brings test-time robustness to corrupted data and helps with calibration. This paper empirically studies the relevance of consistency regularization for training-time robustness to noisy labels. First, we make two interesting and useful observations regarding the consistency of networks trained with the standard cross entropy loss on noisy datasets which are: (i) networks trained on noisy data have lower consistency than those trained on clean data, and(ii) the consistency reduces more significantly around noisy-labelled training data points than correctly-labelled ones. Then, we show that a simple loss function that encourages consistency improves the robustness of the models to label noise on both synthetic (CIFAR-10, CIFAR-100) and real-world (WebVision) noise as well as different noise rates and types and achieves state-of-the-art results.