# Towards Understanding Label Smoothing

arXiv: https://arxiv.org/abs/2006.11653 · HTML: https://mlsanity.com/paper/2006.11653/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2020.

## Built on by (most cited first, 17 of 17)

- [To Smooth or Not? When Label Smoothing Meets Noisy Labels](https://mlsanity.com/paper/2106.04149/) (2021)
- [Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks](https://mlsanity.com/paper/2201.03299/) (2022)
- [Free Lunch for Domain Adversarial Training: Environment Label Smoothing](https://mlsanity.com/paper/2302.00194/) (2023)
- [A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs](https://mlsanity.com/paper/2410.18779/) (2024)
- [Adaptive Label Smoothing To Regularize Large-Scale Graph Training](https://mlsanity.com/paper/2108.13555/) (2021)
- [Improved Visual Fine-tuning with Natural Language Supervision](https://mlsanity.com/paper/2304.01489/) (2023)
- [Data Optimization in Deep Learning: A Survey](https://mlsanity.com/paper/2310.16499/) (2023)
- [How Does Return Distribution in Distributional Reinforcement Learning Help Optimization?](https://mlsanity.com/paper/2209.14513/) (2022)
- [Rethinking Label Smoothing on Multi-hop Question Answering](https://mlsanity.com/paper/2212.09512/) (2022)
- [Jitter: Random Jittering Loss Function](https://mlsanity.com/paper/2106.13749/) (2021)
- [Model Calibration in Dense Classification with Adaptive Label Perturbation](https://mlsanity.com/paper/2307.13539/) (2023)
- [For Better or For Worse? Learning Minimum Variance Features With Label Augmentation](https://mlsanity.com/paper/2402.06855/) (2024)
- [Improving Generalization of Deep Neural Network Acoustic Models with Length Perturbation and N-best Based Label Smoothing](https://mlsanity.com/paper/2203.15176/) (2022)
- [Label Smoothing is Robustification against Model Misspecification](https://mlsanity.com/paper/2305.08501/) (2023)
- [Revisiting the Entropy Semiring for Neural Speech Recognition](https://mlsanity.com/paper/2312.10087/) (2023)
- [Alternating Training-based Label Smoothing Enhances Prompt Generalization](https://mlsanity.com/paper/2508.17846/) (2025)
- [Label Smoothing is a Pragmatic Information Bottleneck](https://mlsanity.com/paper/2508.14077/) (2025)

## Abstract (arXiv)

Label smoothing regularization (LSR) has a great success in training deep neural networks by stochastic algorithms such as stochastic gradient descent and its variants. However, the theoretical understanding of its power from the view of optimization is still rare. This study opens the door to a deep understanding of LSR by initiating the analysis. In this paper, we analyze the convergence behaviors of stochastic gradient descent with label smoothing regularization for solving non-convex problems and show that an appropriate LSR can help to speed up the convergence by reducing the variance. More interestingly, we proposed a simple yet effective strategy, namely Two-Stage LAbel smoothing algorithm (TSLA), that uses LSR in the early training epochs and drops it off in the later training epochs. We observe from the improved convergence result of TSLA that it benefits from LSR in the first stage and essentially converges faster in the second stage. To the best of our knowledge, this is the first work for understanding the power of LSR via establishing convergence complexity of stochastic methods with LSR in non-convex optimization. We empirically demonstrate the effectiveness of the proposed method in comparison with baselines on training ResNet models over benchmark data sets.