# Revisiting Knowledge Distillation via Label Smoothing Regularization

arXiv: https://arxiv.org/abs/1909.11723 · HTML: https://mlsanity.com/paper/1909.11723/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2019.

## Built on by (most cited first, 29 of 29)

- [Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study](https://mlsanity.com/paper/2104.00676/) (2021)
- [Multi-view Subword Regularization](https://mlsanity.com/paper/2103.08490/) (2021)
- [Efficient Knowledge Distillation from Model Checkpoints](https://mlsanity.com/paper/2210.06458/) (2022)
- [In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models](https://mlsanity.com/paper/2212.10670/) (2022)
- [Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones](https://mlsanity.com/paper/2103.05959/) (2021)
- [PrivateSNN: Privacy-Preserving Spiking Neural Networks](https://mlsanity.com/paper/2104.03414/) (2021)
- [Revisiting Knowledge Distillation: An Inheritance and Exploration Framework](https://mlsanity.com/paper/2107.00181/) (2021)
- [Boosting Graph Neural Networks via Adaptive Knowledge Distillation](https://mlsanity.com/paper/2210.05920/) (2022)
- [NCL++: Nested Collaborative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2306.16709/) (2023)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [A Survey on Symbolic Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2408.10210/) (2024)
- [Co-training 2L Submodels for Visual Recognition](https://mlsanity.com/paper/2212.04884/) (2022)
- [SATA: Source Anchoring and Target Alignment Network for Continual Test Time Adaptation](https://mlsanity.com/paper/2304.10113/) (2023)
- [Implicit Regularization of Bregman Proximal Point Algorithm and Mirror Descent on Separable Data](https://mlsanity.com/paper/2108.06808/) (2021)
- [Midpoint Regularization: from High Uncertainty Training to Conservative Classification](https://mlsanity.com/paper/2106.13913/) (2021)
- [Adaptive Pruning of Pretrained Transformer via Differential Inclusions](https://mlsanity.com/paper/2501.03289/) (2025)
- [Knowledge Distillation via Weighted Ensemble of Teaching Assistants](https://mlsanity.com/paper/2206.12005/) (2022)
- [PrUE: Distilling Knowledge from Sparse Teacher Networks](https://mlsanity.com/paper/2207.00586/) (2022)
- [Rich Feature Distillation with Feature Affinity Module for Efficient Image Dehazing](https://mlsanity.com/paper/2207.11250/) (2022)
- [Preserving Fine-Grain Feature Information in Classification via Entropic Regularization](https://mlsanity.com/paper/2208.03684/) (2022)
- [Using Knowledge Distillation to improve interpretable models in a retail banking context](https://mlsanity.com/paper/2209.15496/) (2022)
- [Towards Comparable Knowledge Distillation in Semantic Image Segmentation](https://mlsanity.com/paper/2309.03659/) (2023)
- [Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions](https://mlsanity.com/paper/2607.08164/) (2026)
- [Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation](https://mlsanity.com/paper/2102.07635/) (2021)
- [Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System](https://mlsanity.com/paper/2107.09625/) (2021)
- [Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher](https://mlsanity.com/paper/2410.04140/) (2024)
- [Self-supervised cross-modality learning for uncertainty-aware object detection and recognition in applications which lack pre-labelled training data](https://mlsanity.com/paper/2411.03082/) (2024)
- [A Transformer-in-Transformer Network Utilizing Knowledge Distillation for Image Recognition](https://mlsanity.com/paper/2502.16762/) (2025)
- [Erase to Retain: Low Rank Adaptation Guided Selective Unlearning in Medical Segmentation Networks](https://mlsanity.com/paper/2511.16574/) (2025)

## Abstract (arXiv)

Knowledge Distillation (KD) aims to distill the knowledge of a cumbersome teacher model into a lightweight student model. Its success is generally attributed to the privileged information on similarities among categories provided by the teacher model, and in this sense, only strong teacher models are deployed to teach weaker students in practice. In this work, we challenge this common belief by following experimental observations: 1) beyond the acknowledgment that the teacher can improve the student, the student can also enhance the teacher significantly by reversing the KD procedure; 2) a poorly-trained teacher with much lower accuracy than the student can still improve the latter significantly. To explain these observations, we provide a theoretical analysis of the relationships between KD and label smoothing regularization. We prove that 1) KD is a type of learned label smoothing regularization and 2) label smoothing regularization provides a virtual teacher model for KD. From these results, we argue that the success of KD is not fully due to the similarity information between categories from teachers, but also to the regularization of soft targets, which is equally or even more important. Based on these analyses, we further propose a novel Teacher-free Knowledge Distillation (Tf-KD) framework, where a student model learns from itself or manuallydesigned regularization distribution. The Tf-KD achieves comparable performance with normal KD from a superior teacher, which is well applied when a stronger teacher model is unavailable. Meanwhile, Tf-KD is generic and can be directly deployed for training deep neural networks. Without any extra computation cost, Tf-KD achieves up to 0.65% improvement on ImageNet over well-established baseline models, which is superior to label smoothing regularization.