# To Smooth or Not? When Label Smoothing Meets Noisy Labels

arXiv: https://arxiv.org/abs/2106.04149 · HTML: https://mlsanity.com/paper/2106.04149/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 85 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [How does Disagreement Help Generalization against Label Corruption?](https://mlsanity.com/paper/1901.04215/) (2019)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Combating noisy labels by agreement: A joint training method with co-regularization](https://mlsanity.com/paper/2003.02752/) (2020)
- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [Does label smoothing mitigate label noise?](https://mlsanity.com/paper/2003.02819/) (2020)
- [Learning with Instance-Dependent Label Noise: A Sample Sieve Approach](https://mlsanity.com/paper/2010.02347/) (2020)
- [NLNL: Negative Learning for Noisy Labels](https://mlsanity.com/paper/1908.07387/) (2019)
- [Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates](https://mlsanity.com/paper/1910.03231/) (2019)
- [Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning](https://mlsanity.com/paper/2006.07805/) (2020)
- [Label Distribution Learning](https://mlsanity.com/paper/1408.6027/) (2014)
- [Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective](https://mlsanity.com/paper/2102.00650/) (2021)
- [Rethinking Bias-Variance Trade-off for Generalization of Neural Networks](https://mlsanity.com/paper/2002.11328/) (2020)
- [Towards better decoding and language model integration in sequence to sequence models](https://mlsanity.com/paper/1612.02695/) (2016)
- [Provably Consistent Partial-Label Learning](https://mlsanity.com/paper/2007.08929/) (2020)
- [A Second-Order Approach to Learning with Instance-Dependent Label Noise](https://mlsanity.com/paper/2012.11854/) (2020)

## Built on by (most cited first, 30 of 70)

- [LLM Unlearning via Loss Adjustment with Only Forget Data](https://mlsanity.com/paper/2410.11143/) (2024)
- [Detecting Corrupted Labels Without Training a Model to Predict](https://mlsanity.com/paper/2110.06283/) (2021)
- [Mitigating Memorization of Noisy Labels by Clipping the Model Prediction](https://mlsanity.com/paper/2212.04055/) (2022)
- [RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences](https://mlsanity.com/paper/2402.17257/) (2024)
- [Free Lunch for Domain Adversarial Training: Environment Label Smoothing](https://mlsanity.com/paper/2302.00194/) (2023)
- [FedNoRo: Towards Noise-Robust Federated Learning by Addressing Class Imbalance and Label Noise Heterogeneity](https://mlsanity.com/paper/2305.05230/) (2023)
- [Measuring and Reducing LLM Hallucination without Gold-Standard Answers](https://mlsanity.com/paper/2402.10412/) (2024)
- [To Aggregate or Not? Learning with Separate Noisy Labels](https://mlsanity.com/paper/2206.07181/) (2022)
- [Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Features](https://mlsanity.com/paper/2202.01273/) (2022)
- [Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks](https://mlsanity.com/paper/2309.17002/) (2023)
- [Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Models](https://mlsanity.com/paper/2311.11202/) (2023)
- [Adaptive Integration of Partial Label Learning and Negative Learning for Enhanced Noisy Label Learning](https://mlsanity.com/paper/2312.09505/) (2023)
- [Understanding Self-Distillation in the Presence of Label Noise](https://mlsanity.com/paper/2301.13304/) (2023)
- [Early Stopping Against Label Noise Without Validation Data](https://mlsanity.com/paper/2502.07551/) (2025)
- [Automatic Dataset Construction (ADC): Sample Collection, Data Curation, and Beyond](https://mlsanity.com/paper/2408.11338/) (2024)
- [Weak Proxies are Sufficient and Preferable for Fairness with Missing Sensitive Attributes](https://mlsanity.com/paper/2210.03175/) (2022)
- [Distributionally Robust Post-hoc Classifiers under Prior Shifts](https://mlsanity.com/paper/2309.08825/) (2023)
- [Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models](https://mlsanity.com/paper/2506.12822/) (2025)
- [Be Careful What You Smooth For: Label Smoothing Can Be a Privacy Shield but Also a Catalyst for Model Inversion Attacks](https://mlsanity.com/paper/2310.06549/) (2023)
- [Resurrecting Label Propagation for Graphs with Heterophily and Label Noise](https://mlsanity.com/paper/2310.16560/) (2023)
- [CroSel: Cross Selection of Confident Pseudo Labels for Partial-Label Learning](https://mlsanity.com/paper/2303.10365/) (2023)
- [Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It](https://mlsanity.com/paper/2403.14715/) (2024)
- [Instance-dependent Label Distribution Estimation for Learning with Label Noise](https://mlsanity.com/paper/2212.08380/) (2022)
- [Mitigating Label Noise through Data Ambiguation](https://mlsanity.com/paper/2305.13764/) (2023)
- [SAP: Corrective Machine Unlearning with Scaled Activation Projection for Label Noise Robustness](https://mlsanity.com/paper/2403.08618/) (2024)
- [BoMD: Bag of Multi-label Descriptors for Noisy Chest X-ray Classification](https://mlsanity.com/paper/2203.01937/) (2022)
- [Label Smoothing Improves Machine Unlearning](https://mlsanity.com/paper/2406.07698/) (2024)
- [Foster Adaptivity and Balance in Learning with Noisy Labels](https://mlsanity.com/paper/2407.02778/) (2024)
- [The Importance of Human-Labeled Data in the Era of LLMs](https://mlsanity.com/paper/2306.14910/) (2023)
- [Impact of Noisy Supervision in Foundation Model Learning](https://mlsanity.com/paper/2403.06869/) (2024)

## Abstract (arXiv)

Label smoothing (LS) is an arising learning paradigm that uses the positively weighted average of both the hard training labels and uniformly distributed soft labels. It was shown that LS serves as a regularizer for training data with hard labels and therefore improves the generalization of the model. Later it was reported LS even helps with improving robustness when learning with noisy labels. However, we observed that the advantage of LS vanishes when we operate in a high label noise regime. Intuitively speaking, this is due to the increased entropy of mathbbP(noisy label|X) when the noise rate is high, in which case, further applying LS tends to "over-smooth" the estimated posterior. We proceeded to discover that several learning-with-noisy-labels solutions in the literature instead relate more closely to negative/not label smoothing (NLS), which acts counter to LS and defines as using a negative weight to combine the hard and soft labels! We provide understandings for the properties of LS and NLS when learning with noisy labels. Among other established properties, we theoretically show NLS is considered more beneficial when the label noise rates are high. We provide extensive experimental results on multiple benchmarks to support our findings too. Code is publicly available at https://github.com/UCSC-REAL/negative-label-smoothing.