# Label Smoothing is a Pragmatic Information Bottleneck

arXiv: https://arxiv.org/abs/2508.14077 · HTML: https://mlsanity.com/paper/2508.14077/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 46 of its 60 references that are in the map.
- First posted on arXiv 12 Aug 2025.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.06607/) (2021)
- [Emergence of Invariance and Disentanglement in Deep Representations](https://mlsanity.com/paper/1706.01350/) (2017)
- [Does label smoothing mitigate label noise?](https://mlsanity.com/paper/2003.02819/) (2020)
- [Information Dropout: Learning Optimal Representations Through Noisy Computation](https://mlsanity.com/paper/1611.01353/) (2016)
- [Towards better decoding and language model integration in sequence to sequence models](https://mlsanity.com/paper/1612.02695/) (2016)
- [Delving Deep into Label Smoothing](https://mlsanity.com/paper/2011.12562/) (2020)
- [Invariant Information Bottleneck for Domain Generalization](https://mlsanity.com/paper/2106.06333/) (2021)
- [The Conditional Entropy Bottleneck](https://mlsanity.com/paper/2002.05379/) (2020)
- [Why Do Better Loss Functions Lead to Less Transferable Features?](https://mlsanity.com/paper/2010.16402/) (2020)
- [Nonlinear Information Bottleneck](https://mlsanity.com/paper/1705.02436/) (2017)
- [Efficient Defenses Against Adversarial Attacks](https://mlsanity.com/paper/1707.06728/) (2017)
- [The deterministic information bottleneck](https://mlsanity.com/paper/1604.00268/) (2016)

## Abstract (arXiv)

This study revisits label smoothing via a form of information bottleneck. Under the assumption of sufficient model flexibility and no conflicting labels for the same input, we theoretically and experimentally demonstrate that the model output obtained through label smoothing explores the optimal solution of the information bottleneck. Based on this, label smoothing can be interpreted as a practical approach to the information bottleneck, enabling simple implementation. As an information bottleneck method, we experimentally show that label smoothing also exhibits the property of being insensitive to factors that do not contain information about the target, or to factors that provide no additional information about it when conditioned on another variable.