# Regularization via Structural Label Smoothing

arXiv: https://arxiv.org/abs/2001.01900 · HTML: https://mlsanity.com/paper/2001.01900/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2020.

## Built on by (most cited first, 29 of 29)

- [To Smooth or Not? When Label Smoothing Meets Noisy Labels](https://mlsanity.com/paper/2106.04149/) (2021)
- [Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks](https://mlsanity.com/paper/2201.03299/) (2022)
- [Adaptive Label Smoothing To Regularize Large-Scale Graph Training](https://mlsanity.com/paper/2108.13555/) (2021)
- [Temporal Label Smoothing for Early Event Prediction](https://mlsanity.com/paper/2208.13764/) (2022)
- [MAFA: Managing False Negatives for Vision-Language Pre-training](https://mlsanity.com/paper/2312.06112/) (2023)
- [Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation](https://mlsanity.com/paper/2210.13459/) (2022)
- [Data Optimization in Deep Learning: A Survey](https://mlsanity.com/paper/2310.16499/) (2023)
- [A Gift from Label Smoothing: Robust Training with Adaptive Label Smoothing via Auxiliary Classifier under Label Noise](https://mlsanity.com/paper/2206.07277/) (2022)
- [Rethinking Label Smoothing on Multi-hop Question Answering](https://mlsanity.com/paper/2212.09512/) (2022)
- [How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?](https://mlsanity.com/paper/2510.17526/) (2025)
- [Midpoint Regularization: from High Uncertainty Training to Conservative Classification](https://mlsanity.com/paper/2106.13913/) (2021)
- [Probabilistic Approach for Road-Users Detection](https://mlsanity.com/paper/2112.01360/) (2021)
- [For Better or For Worse? Learning Minimum Variance Features With Label Augmentation](https://mlsanity.com/paper/2402.06855/) (2024)
- [Confidence Calibration for Intent Detection via Hyperspherical Space and Rebalanced Accuracy-Uncertainty Loss](https://mlsanity.com/paper/2203.09278/) (2022)
- [Label Smoothing is Robustification against Model Misspecification](https://mlsanity.com/paper/2305.08501/) (2023)
- [Robust Classification by Coupling Data Mollification with Label Smoothing](https://mlsanity.com/paper/2406.01494/) (2024)
- [A Tutorial on Clinical Speech AI Development: From Data Collection to Model Validation](https://mlsanity.com/paper/2410.21640/) (2024)
- [Regularization of ML models for Earth systems by using longer model timesteps](https://mlsanity.com/paper/2503.18023/) (2025)
- [Heatmap Regression without Soft-Argmax for Facial Landmark Detection](https://mlsanity.com/paper/2508.14929/) (2025)
- [Selective Output Smoothing Regularization: Regularize Neural Networks by Softening Output Distributions](https://mlsanity.com/paper/2103.15383/) (2021)
- [Disturbing Target Values for Neural Network Regularization](https://mlsanity.com/paper/2110.05003/) (2021)
- [Smoothly Giving up: Robustness for Simple Models](https://mlsanity.com/paper/2302.09114/) (2023)
- [Improving Generalization in Meta-Learning via Meta-Gradient Augmentation](https://mlsanity.com/paper/2306.08460/) (2023)
- [Revisiting the Role of Label Smoothing in Enhanced Text Sentiment Classification](https://mlsanity.com/paper/2312.06522/) (2023)
- [Unsupervised Spatial-Temporal Feature Enrichment and Fidelity Preservation Network for Skeleton based Action Recognition](https://mlsanity.com/paper/2401.14034/) (2024)
- [Layer-wise Regularized Dropout for Neural Language Models](https://mlsanity.com/paper/2402.16361/) (2024)
- [Posterior Label Smoothing for Node Classification](https://mlsanity.com/paper/2406.00410/) (2024)
- [Generalizing Few Data to Unseen Domains Flexibly Based on Label Smoothing Integrated with Distributionally Robust Optimization](https://mlsanity.com/paper/2408.05082/) (2024)
- [Label Smoothing is a Pragmatic Information Bottleneck](https://mlsanity.com/paper/2508.14077/) (2025)

## Abstract (arXiv)

Regularization is an effective way to promote the generalization performance of machine learning models. In this paper, we focus on label smoothing, a form of output distribution regularization that prevents overfitting of a neural network by softening the ground-truth labels in the training data in an attempt to penalize overconfident outputs. Existing approaches typically use cross-validation to impose this smoothing, which is uniform across all training data. In this paper, we show that such label smoothing imposes a quantifiable bias in the Bayes error rate of the training data, with regions of the feature space with high overlap and low marginal likelihood having a lower bias and regions of low overlap and high marginal likelihood having a higher bias. These theoretical results motivate a simple objective function for data-dependent smoothing to mitigate the potential negative consequences of the operation while maintaining its desirable properties as a regularizer. We call this approach Structural Label Smoothing (SLS). We implement SLS and empirically validate on synthetic, Higgs, SVHN, CIFAR-10, and CIFAR-100 datasets. The results confirm our theoretical insights and demonstrate the effectiveness of the proposed method in comparison to traditional label smoothing.