mlsanityData to 14 Oct 2026

Label Smoothing is a Pragmatic Information Bottleneck

arXiv 2508.14077 ↗First posted 12 Aug 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

This study revisits label smoothing via a form of information bottleneck. Under the assumption of sufficient model flexibility and no conflicting labels for the same input, we theoretically and experimentally demonstrate that the model output obtained through label smoothing explores the optimal solution of the information bottleneck. Based on this, label smoothing can be interpreted as a practical approach to the information bottleneck, enabling simple implementation. As an information bottleneck method, we experimentally show that label smoothing also exhibits the property of being insensitive to factors that do not contain information about the target, or to factors that provide no additional information about it when conditioned on another variable.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 46 of its 60 references that are in the map. First posted on arXiv 12 Aug 2025.

Builds on

46 of 60 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2022A ConvNet for the 2020scited by 3,579
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  10. 2019When Does Label Smoothing Help?cited by 988
  11. 2016Deep Variational Information Bottleneckcited by 930
  12. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
34 more
  1. 2018Recognition in Terra Incognitacited by 658
  2. 2019On Variational Bounds of Mutual Informationcited by 559
  3. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  4. 2014Recurrent Models of Visual Attentioncited by 349
  5. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 264
  6. 2019Human uncertainty makes classification more robustcited by 235
  7. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  8. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 215
  9. 2020Does label smoothing mitigate label noise?cited by 204
  10. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 132
  11. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  12. 2020Delving Deep into Label Smoothingcited by 94
  13. 2021Invariant Information Bottleneck for Domain Generalizationcited by 86
  14. 2020The Conditional Entropy Bottleneckcited by 75
  15. 2020Why Do Better Loss Functions Lead to Less Transferable Features?cited by 73
  16. 2017Nonlinear Information Bottleneckcited by 71
  17. 2017Efficient Defenses Against Adversarial Attackscited by 53
  18. 2016The deterministic information bottleneckcited by 42
  19. 2020Generalized Entropy Regularization or: There's Nothing Special about Label Smoothingcited by 41
  20. 2018Uncertainty in the Variational Information Bottleneckcited by 39
  21. 2018Caveats for information bottleneck in deterministic scenarioscited by 34
  22. 2020Regularization via Structural Label Smoothingcited by 29
  23. 2021PAC-Bayes Information Bottleneckcited by 28
  24. 2021Diversifying Dialog Generation via Adaptive Label Smoothingcited by 25
  25. 2020Disentangled Information Bottleneckcited by 24
  26. 2021Understanding Instance-Level Label Noise: Disparate Impacts and Treatmentscited by 22
  27. 2021Deep Deterministic Information Bottleneck with Matrix-based Entropy Functionalcited by 18
  28. 2020Towards Understanding Label Smoothingcited by 17
  29. 2020The Dual Information Bottleneckcited by 14
  30. 2020An Investigation of how Label Smoothing Affects Generalizationcited by 11
  31. 2024Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix Itcited by 11
  32. 2020The Role of Mutual Information in Variational Classifierscited by 3
  33. 2024For Better or For Worse? Learning Minimum Variance Features With Label Augmentationcited by 3
  34. 2024Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Trainingcited by 1

Built on by

Not cited yet.