# Label Smoothing is Robustification against Model Misspecification

arXiv: https://arxiv.org/abs/2305.08501 · HTML: https://mlsanity.com/paper/2305.08501/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 15 May 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Adversarial Logit Pairing](https://mlsanity.com/paper/1803.06373/) (2018)
- [Does label smoothing mitigate label noise?](https://mlsanity.com/paper/2003.02819/) (2020)
- [Towards better decoding and language model integration in sequence to sequence models](https://mlsanity.com/paper/1612.02695/) (2016)
- [Distributional Smoothing with Virtual Adversarial Training](https://mlsanity.com/paper/1507.00677/) (2015)
- [Generalized Entropy Regularization or: There's Nothing Special about Label Smoothing](https://mlsanity.com/paper/2005.00820/) (2020)
- [Evaluating and Understanding the Robustness of Adversarial Logit Pairing](https://mlsanity.com/paper/1807.10272/) (2018)
- [Regularization via Structural Label Smoothing](https://mlsanity.com/paper/2001.01900/) (2020)
- [Towards Understanding Label Smoothing](https://mlsanity.com/paper/2006.11653/) (2020)
- [Cost-Sensitive Regularization for Diabetic Retinopathy Grading from Eye Fundus Images](https://mlsanity.com/paper/2010.00291/) (2020)
- [Adversarial Robustness via Label-Smoothing](https://mlsanity.com/paper/1906.11567/) (2019)

## Built on by (most cited first, 2 of 2)

- [Isotonic Bradley-Terry Model for Paired Comparison Data](https://mlsanity.com/paper/2608.02081/) (2026)
- [Unimodality-Promoting Regularized Learning for Ordinal Regression](https://mlsanity.com/paper/2608.08359/) (2026)

## Abstract (arXiv)

Label smoothing (LS) adopts smoothed targets in classification tasks. For example, in binary classification, instead of the one-hot target (1,0)^top used in conventional logistic regression (LR), LR with LS (LSLR) uses the smoothed target (1-fracα2,fracα2)^top with a smoothing level α∈(0,1), which causes squeezing of values of the logit. Apart from the common regularization-based interpretation of LS that leads to an inconsistent probability estimator, we regard LSLR as modifying the loss function and consistent estimator for probability estimation. In order to study the significance of each of these two modifications by LSLR, we introduce a modified LSLR (MLSLR) that uses the same loss function as LSLR and the same consistent estimator as LR, while not squeezing the logits. For the loss function modification, we theoretically show that MLSLR with a larger smoothing level has lower efficiency with correctly-specified models, while it exhibits higher robustness against model misspecification than LR. Also, for the modification of the probability estimator, an experimental comparison between LSLR and MLSLR showed that this modification and squeezing of the logits in LSLR have negative effects on the probability estimation and classification performance. The understanding of the properties of LS provided by these comparisons allows us to propose MLSLR as an improvement over LSLR.