mlsanityData to 7 Oct 2026

To Smooth or Not? When Label Smoothing Meets Noisy Labels

arXiv 2106.04149 ↗First posted 8 Jun 2021cs.LGcited by 70

Not on the map. Why?

Abstract (arXiv)

Label smoothing (LS) is an arising learning paradigm that uses the positively weighted average of both the hard training labels and uniformly distributed soft labels. It was shown that LS serves as a regularizer for training data with hard labels and therefore improves the generalization of the model. Later it was reported LS even helps with improving robustness when learning with noisy labels. However, we observed that the advantage of LS vanishes when we operate in a high label noise regime. Intuitively speaking, this is due to the increased entropy of mathbbP(noisy label|X) when the noise rate is high, in which case, further applying LS tends to "over-smooth" the estimated posterior. We proceeded to discover that several learning-with-noisy-labels solutions in the literature instead relate more closely to negative/not label smoothing (NLS), which acts counter to LS and defines as using a negative weight to combine the hard and soft labels! We provide understandings for the properties of LS and NLS when learning with noisy labels. Among other established properties, we theoretically show NLS is considered more beneficial when the label noise rates are high. We provide extensive experimental results on multiple benchmarks to support our findings too. Code is publicly available at https://github.com/UCSC-REAL/negative-label-smoothing.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 51 of its 85 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

51 of 85 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  8. 2019When Does Label Smoothing Help?cited by 985
  9. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  10. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  11. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
  12. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
38 more
  1. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 414
  2. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 411
  3. 2019How does Disagreement Help Generalization against Label Corruption?cited by 390
  4. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  5. 2020Combating noisy labels by agreement: A joint training method with co-regularizationcited by 289
  6. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 259
  7. 2020Normalized Loss Functions for Deep Learning with Noisy Labelscited by 257
  8. 2020Does label smoothing mitigate label noise?cited by 204
  9. 2020Learning with Instance-Dependent Label Noise: A Sample Sieve Approachcited by 158
  10. 2019NLNL: Negative Learning for Noisy Labelscited by 151
  11. 2019Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Ratescited by 142
  12. 2020Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learningcited by 142
  13. 2014Label Distribution Learningcited by 122
  14. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  15. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  16. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  17. 2020Provably Consistent Partial-Label Learningcited by 92
  18. 2020A Second-Order Approach to Learning with Instance-Dependent Label Noisecited by 90
  19. 2017Learning from Complementary Labelscited by 88
  20. 2021Sample Selection with Uncertainty of Losses for Learning with Noisy Labelscited by 79
  21. 2021Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labelscited by 74
  22. 2013Classification with Asymmetric Label Noise: Consistency and Maximal Denoisingcited by 67
  23. 2016Mixture Proportion Estimation via Kernel Embedding of Distributionscited by 67
  24. 2020Fair Classification with Group-Dependent Label Noisecited by 67
  25. 2021Clusterability as an Alternative to Anchor Points When Learning with Noisy Labelscited by 65
  26. 2020Confidence Scores Make Instance-dependent Label-noise Learning Possiblecited by 63
  27. 2019Robust Bi-Tempered Logistic Loss Based on Bregman Divergencescited by 61
  28. 2018Complementary-Label Learning for Arbitrary Losses and Modelscited by 57
  29. 2020When Optimizing f-divergence is Robust with Label Noisecited by 44
  30. 2020Regularization via Structural Label Smoothingcited by 29
  31. 2019L_DMI: An Information-theoretic Noise-robust Loss Functioncited by 26
  32. 2021Understanding Instance-Level Label Noise: Disparate Impacts and Treatmentscited by 22
  33. 2020Towards Understanding Label Smoothingcited by 17
  34. 2021Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyondcited by 16
  35. 2021Convolutional Normalization: Improving Deep Convolutional Network Robustness and Trainingcited by 15
  36. 2020An Investigation of how Label Smoothing Affects Generalizationcited by 11
  37. 2021Instance Correction for Learning with Open-set Noisy Labelscited by 7
  38. 2020Extended T: Learning with Mixed Closed-set and Open-set Noisy Labelscited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 17; 2024: 19; 2025: 20; 2026: 4.

70 papers.

Sort:

By area:

  1. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  2. 2021Detecting Corrupted Labels Without Training a Model to Predictcited by 57
  3. 2022Mitigating Memorization of Noisy Labels by Clipping the Model Predictioncited by 40
  4. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
  5. 2023Free Lunch for Domain Adversarial Training: Environment Label Smoothingcited by 33
  6. 2023FedNoRo: Towards Noise-Robust Federated Learning by Addressing Class Imbalance and Label Noise Heterogeneitycited by 33
  7. 2024Measuring and Reducing LLM Hallucination without Gold-Standard Answerscited by 31
  8. 2022To Aggregate or Not? Learning with Separate Noisy Labelscited by 29
  9. 2022Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Featurescited by 27
  10. 2023Understanding and Mitigating the Label Noise in Pre-training on Downstream Taskscited by 27
  11. 2023Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Modelscited by 26
  12. 2023Adaptive Integration of Partial Label Learning and Negative Learning for Enhanced Noisy Label Learningcited by 22
38 more
  1. 2025Early Stopping Against Label Noise Without Validation Datacited by 21
  2. 2023Understanding Self-Distillation in the Presence of Label Noisecited by 20
  3. 2024Automatic Dataset Construction (ADC): Sample Collection, Data Curation, and Beyondcited by 18
  4. 2022Weak Proxies are Sufficient and Preferable for Fairness with Missing Sensitive Attributescited by 17
  5. 2023Distributionally Robust Post-hoc Classifiers under Prior Shiftscited by 16
  6. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  7. 2023Be Careful What You Smooth For: Label Smoothing Can Be a Privacy Shield but Also a Catalyst for Model Inversion Attackscited by 12
  8. 2023Resurrecting Label Propagation for Graphs with Heterophily and Label Noisecited by 12
  9. 2023CroSel: Cross Selection of Confident Pseudo Labels for Partial-Label Learningcited by 11
  10. 2024Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix Itcited by 11
  11. 2023Mitigating Label Noise through Data Ambiguationcited by 10
  12. 2024SAP: Corrective Machine Unlearning with Scaled Activation Projection for Label Noise Robustnesscited by 10
  13. 2022BoMD: Bag of Multi-label Descriptors for Noisy Chest X-ray Classificationcited by 9
  14. 2022Instance-dependent Label Distribution Estimation for Learning with Label Noisecited by 9
  15. 2024Foster Adaptivity and Balance in Learning with Noisy Labelscited by 9
  16. 2023The Importance of Human-Labeled Data in the Era of LLMscited by 8
  17. 2024Impact of Noisy Supervision in Foundation Model Learningcited by 8
  18. 2024Skeleton-Based Human Action Recognition with Noisy Labelscited by 8
  19. 2024Label Smoothing Improves Machine Unlearningcited by 8
  20. 2025Calibrated Language Models and How to Find Them with Label Smoothingcited by 7
  21. 2024Mitigating Label Noise on Graph via Topological Sample Selectioncited by 6
  22. 2024Mitigating Label Noise using Prompt-Based Hyperbolic Meta-Learning in Open-Set Domain Generalizationcited by 6
  23. 2025Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoringcited by 6
  24. 2025ε-Softmax: Approximating One-Hot Vectors for Mitigating Label Noisecited by 6
  25. 2025When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identificationcited by 5
  26. 2025SelectMix: Enhancing Label Noise Robustness through Targeted Sample Mixingcited by 5
  27. 2022A Gift from Label Smoothing: Robust Training with Adaptive Label Smoothing via Auxiliary Classifier under Label Noisecited by 4
  28. 2023Leveraging an Alignment Set in Tackling Instance-Dependent Label Noisecited by 4
  29. 2024For Better or For Worse? Learning Minimum Variance Features With Label Augmentationcited by 3
  30. 2025Noise-Resilient Point-wise Anomaly Detection in Time Series Using Weak Segment Labelscited by 3
  31. 2025Enhanced Sample Selection with Confidence Tracking: Identifying Correctly Labeled yet Hard-to-Learn Samples in Noisy Datacited by 3
  32. 2025LM-mixup: Text Data Augmentation via Language Model based Mixupcited by 3
  33. 2025Label Smoothing Improves Gradient Ascent in LLM Unlearningcited by 3
  34. 2026Optimal Transport for LLM Reward Modeling from Noisy Preferencecited by 3
  35. 2023Rethinking Noisy Label Learning in Real-world Annotation Scenarios from the Noise-type Perspectivecited by 2
  36. 2025Robust Classification with Noisy Labels Based on Posterior Maximizationcited by 2
  37. 2025Performance Estimation in Binary Classification Using Calibrated Confidencecited by 2
  38. 2022Propagating Variational Model Uncertainty for Bioacoustic Call Label Smoothingcited by 1

50 most cited of 70 All 70 (JSON)