# Delving Deep into Label Smoothing

arXiv: https://arxiv.org/abs/2011.12562 · HTML: https://mlsanity.com/paper/2011.12562/

## Key facts

- Cited by 94 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2020.

## Built on by (most cited first, 30 of 94)

- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels](https://mlsanity.com/paper/2303.13005/) (2023)
- [Eliciting and Learning with Soft Labels from Every Annotator](https://mlsanity.com/paper/2207.00810/) (2022)
- [Towards Calibrated Robust Fine-Tuning of Vision-Language Models](https://mlsanity.com/paper/2311.01723/) (2023)
- [Self-Distillation from the Last Mini-Batch for Consistency Regularization](https://mlsanity.com/paper/2203.16172/) (2022)
- [Salient Objects in Clutter](https://mlsanity.com/paper/2105.03053/) (2021)
- [Relieving Long-tailed Instance Segmentation via Pairwise Class Balance](https://mlsanity.com/paper/2201.02784/) (2022)
- [ZLPR: A Novel Loss for Multi-label Classification](https://mlsanity.com/paper/2208.02955/) (2022)
- [SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Models](https://mlsanity.com/paper/2411.02433/) (2024)
- [Speechformer: Reducing Information Loss in Direct Speech Translation](https://mlsanity.com/paper/2109.04574/) (2021)
- [Event Transformer+. A multi-purpose solution for efficient event data processing](https://mlsanity.com/paper/2211.12222/) (2022)
- [Efficient One Pass Self-distillation with Zipf's Label Smoothing](https://mlsanity.com/paper/2207.12980/) (2022)
- [Soft Augmentation for Image Classification](https://mlsanity.com/paper/2211.04625/) (2022)
- [BoMD: Bag of Multi-label Descriptors for Noisy Chest X-ray Classification](https://mlsanity.com/paper/2203.01937/) (2022)
- [It Is All About Data: A Survey on the Effects of Data on Adversarial Robustness](https://mlsanity.com/paper/2303.09767/) (2023)
- [AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels](https://mlsanity.com/paper/2410.22506/) (2024)
- [Open-set Text Recognition via Character-Context Decoupling](https://mlsanity.com/paper/2204.05535/) (2022)
- [Regroup Median Loss for Combating Label Noise](https://mlsanity.com/paper/2312.06273/) (2023)
- [Noisy Label Processing for Classification: A Survey](https://mlsanity.com/paper/2404.04159/) (2024)
- [DynamicKD: An Effective Knowledge Distillation via Dynamic Entropy Correction-Based Distillation for Gap Optimizing](https://mlsanity.com/paper/2305.05233/) (2023)
- [Navigation-Oriented Scene Understanding for Robotic Autonomy: Learning to Segment Driveability in Egocentric Images](https://mlsanity.com/paper/2109.07245/) (2021)
- [Inverse Image Frequency for Long-tailed Image Recognition](https://mlsanity.com/paper/2209.04861/) (2022)
- [LUMix: Improving Mixup by Better Modelling Label Uncertainty](https://mlsanity.com/paper/2211.15846/) (2022)
- [Data Optimization in Deep Learning: A Survey](https://mlsanity.com/paper/2310.16499/) (2023)
- [Convolutional Fine-Grained Classification with Self-Supervised Target Relation Regularization](https://mlsanity.com/paper/2208.01997/) (2022)
- [In and Out-of-Domain Text Adversarial Robustness via Label Smoothing](https://mlsanity.com/paper/2212.10258/) (2022)
- [Structure-Aware Label Smoothing for Graph Neural Networks](https://mlsanity.com/paper/2112.00499/) (2021)
- [Data-Driven Estimation of the False Positive Rate of the Bayes Binary Classifier via Soft Labels](https://mlsanity.com/paper/2401.15500/) (2024)
- [MaxSup: Overcoming Representation Collapse in Label Smoothing](https://mlsanity.com/paper/2502.15798/) (2025)
- [Jitter: Random Jittering Loss Function](https://mlsanity.com/paper/2106.13749/) (2021)

## Abstract (arXiv)

Label smoothing is an effective regularization tool for deep neural networks (DNNs), which generates soft labels by applying a weighted average between the uniform distribution and the hard label. It is often used to reduce the overfitting problem of training DNNs and further improve classification performance. In this paper, we aim to investigate how to generate more reliable soft labels. We present an Online Label Smoothing (OLS) strategy, which generates soft labels based on the statistics of the model prediction for the target category. The proposed OLS constructs a more reasonable probability distribution between the target categories and non-target categories to supervise DNNs. Experiments demonstrate that based on the same classification models, the proposed approach can effectively improve the classification performance on CIFAR-100, ImageNet, and fine-grained datasets. Additionally, the proposed method can significantly improve the robustness of DNN models to noisy labels compared to current label smoothing approaches.