mlsanityData to 13 Oct 2026

Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study

arXiv 2104.00676 ↗First posted 1 Apr 2021cs.LG cs.AI cs.CLcited by 50

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

This work aims to empirically clarify a recently discovered perspective that label smoothing is incompatible with knowledge distillation. We begin by introducing the motivation behind on how this incompatibility is raised, i.e., label smoothing erases relative information between teacher logits. We provide a novel connection on how label smoothing affects distributions of semantically similar and dissimilar classes. Then we propose a metric to quantitatively measure the degree of erased information in sample's representation. After that, we study its one-sidedness and imperfection of the incompatibility view through massive analyses, visualizations and comprehensive experiments on Image Classification, Binary Networks, and Neural Machine Translation. Finally, we broadly discuss several circumstances wherein label smoothing will indeed lose its effectiveness. Project page: http://zhiqiangshen.com/projects/LS_and_KD/index.html.

Key facts. Cited by 50 papers in the map (data through 13 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 36 of its 43 references that are in the map. First posted on arXiv 1 Apr 2021.

Builds on

36 of 43 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  5. 2016Densely Connected Convolutional Networkscited by 6,707
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,457
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,786
  8. 2016Wide Residual Networkscited by 3,063
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,882
  10. 2020Supervised Contrastive Learningcited by 2,811
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,544
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,631
24 more
  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,488
  2. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,306
  3. 2019When Does Label Smoothing Help?cited by 986
  4. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  5. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  6. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 462
  7. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  8. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 235
  9. 2019Fixing the train-test resolution discrepancycited by 209
  10. 2020Does label smoothing mitigate label noise?cited by 204
  11. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  12. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 184
  13. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 122
  14. 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 111
  15. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  16. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  17. 2020Understanding and Improving Knowledge Distillationcited by 83
  18. 2018Label Refinery: Improving ImageNet Classification through Label Progressioncited by 65
  19. 2020Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learningcited by 62
  20. 2018MEAL: Multi-Model Ensemble via Adversarial Learningcited by 52
  21. 2020MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Trickscited by 40
  22. 2019Revisiting Knowledge Distillation via Label Smoothing Regularizationcited by 29
  23. 2020Building a Multi-domain Neural Machine Translation Model using Knowledge Distillationcited by 9
  24. 2019Noise as a Resource for Learning in Knowledge Distillationcited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 6; 2022: 14; 2023: 17; 2024: 7; 2025: 5; 2026: 1.

50 papers.

Sort:

By area:

  1. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 426
  2. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  3. 2022Knowledge Distillation from A Stronger Teachercited by 204
  4. 2022Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generationcited by 67
  5. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  6. 2021A Fast Knowledge Distillation Framework for Visual Recognitioncited by 50
  7. 2023Rethinking Confidence Calibration for Failure Predictioncited by 43
  8. 2023Re-thinking Model Inversion Attacks Against Deep Neural Networkscited by 39
  9. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  10. 2021Sliced Recursive Transformercited by 26
  11. 2021Network Augmentation for Tiny Deep Learningcited by 24
  12. 2022Asymmetric Temperature Scaling Makes Larger Networks Teach Well Againcited by 23
38 more
  1. 2022Meta Knowledge Distillationcited by 20
  2. 2022Knowledge Condensation Distillationcited by 20
  3. 2021Class-Distribution-Aware Calibration for Long-Tailed Visual Recognitioncited by 19
  4. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 19
  5. 2022Respecting Transfer Gap in Knowledge Distillationcited by 18
  6. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  7. 2023Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcementcited by 11
  8. 2023Student-friendly Knowledge Distillationcited by 11
  9. 2024Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix Itcited by 11
  10. 2024Revisiting Confidence Estimation: Towards Reliable Failure Predictioncited by 9
  11. 2021Data-Free Neural Architecture Search via Recursive Label Calibrationcited by 8
  12. 2023On the Impact of Knowledge Distillation for Model Interpretabilitycited by 8
  13. 2023Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillationcited by 7
  14. 2023DOT: A Distillation-Oriented Trainercited by 7
  15. 2023Heterogeneous Continual Learningcited by 6
  16. 2024MergeNet: Knowledge Migration across Heterogeneous Models, Tasks, and Modalitiescited by 6
  17. 2022Understanding the Role of Mixup in Knowledge Distillation: An Empirical Studycited by 5
  18. 2023FerKD: Surgical Label Adaptation for Efficient Distillationcited by 5
  19. 2024Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatchcited by 4
  20. 2024Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillationcited by 4
  21. 2025MaxSup: Overcoming Representation Collapse in Label Smoothingcited by 4
  22. 2025Local Dense Logit Relations for Enhanced Knowledge Distillationcited by 4
  23. 2023Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy?cited by 3
  24. 2024Improving Data Augmentation for Robust Visual Question Answering with Effective Curriculum Learningcited by 2
  25. 2022Improving the Robustness and Generalization of Deep Neural Network with Confidence Threshold Reductioncited by 1
  26. 2022PrUE: Distilling Knowledge from Sparse Teacher Networkscited by 1
  27. 2022Preserving Fine-Grain Feature Information in Classification via Entropic Regularizationcited by 1
  28. 2023Open-Set RF Fingerprinting via Improved Prototype Learningcited by 1
  29. 2024Enhancing Suicide Risk Detection on Social Media through Semi-Supervised Deep Label Smoothingcited by 1
  30. 2025Overcoming Language Priors for Visual Question Answering Based on Knowledge Distillationcited by 1
  31. 2026The Implicit Bias of Logit Regularizationcited by 1
  32. 2022PEA: Improving the Performance of ReLU Networks for Free by Using Progressive Ensemble Activationscited by 0
  33. 2022Symbolic Distillation for Learned TCP Congestion Controlcited by 0
  34. 2023Rethinking Soft Label in Label Distribution Learning Perspectivecited by 0
  35. 2023LABO: Towards Learning Optimal Label Regularization via Bi-level Optimizationcited by 0
  36. 2023Towards Better Query Classification with Multi-Expert Knowledge Condensation in JD Ads Searchcited by 0
  37. 2025DisTaC: Conditioning Task Vectors via Distillation for Robust Model Mergingcited by 0
  38. 2025iCD: A Implicit Clustering Distillation Mathod for Structural Information Miningcited by 0