Data to 7 Oct 2026

When Does Label Smoothing Help?

arXiv 1906.02629 ↗First posted 6 Jun 2019cs.LG stat.MLcited by 985

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels. Smoothing the labels in this way prevents the network from becoming over-confident and label smoothing has been used in many state-of-the-art models, including image classification, language translation and speech recognition. Despite its widespread use, label smoothing is still poorly understood. Here we show empirically that in addition to improving generalization, label smoothing improves model calibration which can significantly improve beam-search. However, we also observe that if a teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective. To explain these observations, we visualize how label smoothing changes the representations learned by the penultimate layer of the network. We show that label smoothing encourages the representations of training examples from the same class to group in tight clusters. This results in loss of information in the logits about resemblances between instances of different classes, which is necessary for distillation, but does not hurt generalization or calibration of the model's predictions.

Key facts. Cited by 985 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 169; 2022: 211; 2023: 211; 2024: 157; 2025: 128; 2026: 108.

985 papers.

Sort:

By area:

2021Robust fine-tuning of zero-shot modelscited by 785
2021A Survey of Uncertainty in Deep Neural Networkscited by 675
2021Revisiting the Calibration of Modern Neural Networkscited by 387
2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
2022Large Language Models Are Reasoning Teacherscited by 318
2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
2023Large Language Model Unlearningcited by 288
2022Masked Autoencoders that Listencited by 271
2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 251
2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
2021Improving Calibration for Long-Tailed Recognitioncited by 212
2022Visual Attention Networkcited by 210

38 more

2023Better Diffusion Models Further Improve Adversarial Trainingcited by 207
2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 177
2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
2023Getting aligned on representational alignmentcited by 142
2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
2021Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extractioncited by 129
2021Unbalanced minibatch Optimal Transport; applications to Domain Adaptationcited by 123
2023EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularizationcited by 123
2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
2021Source-free Domain Adaptation via Avatar Prototype Generation and Adaptationcited by 111
2021A Survey on Data Augmentation for Text Classificationcited by 111
2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 95
2021The Devil is in the Margin: Margin-based Label Smoothing for Network Calibrationcited by 91
2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
2021Soft Calibration Objectives for Neural Networkscited by 89
2021AugMax: Adversarial Composition of Random Augmentations for Robust Trainingcited by 89
2022Physics-Informed Machine Learning: A Survey on Problems, Methods and Applicationscited by 88
2024LAA-Net: Localized Artifact Attention Network for Quality-Agnostic and Generalizable Deepfake Detectioncited by 88
2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
2021Learning Neural Network Subspacescited by 82
2021Multi-Label Learning from Single Positive Labelscited by 81
2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
2023Fine-Grained Visual Promptingcited by 81
2022Source-Free Unsupervised Domain Adaptation: A Surveycited by 79
2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78

50 most cited of 985 All 985 (JSON)