# Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students

arXiv: https://arxiv.org/abs/1805.05551 · HTML: https://mlsanity.com/paper/1805.05551/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 May 2018.

## Built on by (most cited first, 30 of 50)

- [Class-Incremental Continual Learning into the eXtended DER-verse](https://mlsanity.com/paper/2201.00766/) (2022)
- [Learning Affordance Grounding from Exocentric Images](https://mlsanity.com/paper/2203.09905/) (2022)
- [Robust Preference Optimization through Reward Model Distillation](https://mlsanity.com/paper/2405.19316/) (2024)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Knowledge Inheritance for Pre-trained Language Models](https://mlsanity.com/paper/2105.13880/) (2021)
- [Efficient Knowledge Distillation from Model Checkpoints](https://mlsanity.com/paper/2210.06458/) (2022)
- [Multimodal Knowledge Expansion](https://mlsanity.com/paper/2103.14431/) (2021)
- [Distilling Holistic Knowledge with Graph Neural Networks](https://mlsanity.com/paper/2108.05507/) (2021)
- [Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information](https://mlsanity.com/paper/2401.08732/) (2024)
- [RomeBERT: Robust Training of Multi-Exit BERT](https://mlsanity.com/paper/2101.09755/) (2021)
- [Compressing Deep Graph Neural Networks via Adversarial Knowledge Distillation](https://mlsanity.com/paper/2205.11678/) (2022)
- [One Timestep is All You Need: Training Spiking Neural Networks with Ultra Low Latency](https://mlsanity.com/paper/2110.05929/) (2021)
- [Understanding the Gains from Repeated Self-Distillation](https://mlsanity.com/paper/2407.04600/) (2024)
- [Disentangle and Remerge: Interventional Knowledge Distillation for Few-Shot Object Detection from A Conditional Causal Perspective](https://mlsanity.com/paper/2208.12681/) (2022)
- [How to Train the Teacher Model for Effective Knowledge Distillation](https://mlsanity.com/paper/2407.18041/) (2024)
- [Crowd Localization from Gaussian Mixture Scoped Knowledge and Scoped Teacher](https://mlsanity.com/paper/2206.05717/) (2022)
- [FS-BAN: Born-Again Networks for Domain Generalization Few-Shot Classification](https://mlsanity.com/paper/2208.10930/) (2022)
- [Self-Distillation for Further Pre-training of Transformers](https://mlsanity.com/paper/2210.02871/) (2022)
- [Language Modelling via Learning to Rank](https://mlsanity.com/paper/2110.06961/) (2021)
- [MISSU: 3D Medical Image Segmentation via Self-distilling TransUNet](https://mlsanity.com/paper/2206.00902/) (2022)
- [Robust Distillation for Worst-class Performance](https://mlsanity.com/paper/2206.06479/) (2022)
- [Boosting the Adversarial Transferability of Surrogate Models with Dark Knowledge](https://mlsanity.com/paper/2206.08316/) (2022)
- [Simulated Annealing in Early Layers Leads to Better Generalization](https://mlsanity.com/paper/2304.04858/) (2023)
- [Role of Data Augmentation Strategies in Knowledge Distillation for Wearable Sensor Data](https://mlsanity.com/paper/2201.00111/) (2022)
- [Decentralized Learning with Multi-Headed Distillation](https://mlsanity.com/paper/2211.15774/) (2022)
- [Isotonic Data Augmentation for Knowledge Distillation](https://mlsanity.com/paper/2107.01412/) (2021)
- [Knowledge Distillation-based Information Sharing for Online Process Monitoring in Decentralized Manufacturing System](https://mlsanity.com/paper/2302.12004/) (2023)
- [Distilled Transformers with Locally Enhanced Global Representations for Face Forgery Detection](https://mlsanity.com/paper/2412.20156/) (2024)
- [Midpoint Regularization: from High Uncertainty Training to Conservative Classification](https://mlsanity.com/paper/2106.13913/) (2021)
- [Deep Neural Compression Via Concurrent Pruning and Self-Distillation](https://mlsanity.com/paper/2109.15014/) (2021)

## Abstract (arXiv)

We focus on the problem of training a deep neural network in generations. The flowchart is that, in order to optimize the target network (student), another network (teacher) with the same architecture is first trained, and used to provide part of supervision signals in the next stage. While this strategy leads to a higher accuracy, many aspects (e.g., why teacher-student optimization helps) still need further explorations. This paper studies this problem from a perspective of controlling the strictness in training the teacher network. Existing approaches mostly used a hard distribution (e.g., one-hot vectors) in training, leading to a strict teacher which itself has a high accuracy, but we argue that the teacher needs to be more tolerant, although this often implies a lower accuracy. The implementation is very easy, with merely an extra loss term added to the teacher network, facilitating a few secondary classes to emerge and complement to the primary class. Consequently, the teacher provides a milder supervision signal (a less peaked distribution), and makes it possible for the student to learn from inter-class similarity and potentially lower the risk of over-fitting. Experiments are performed on standard image classification tasks (CIFAR100 and ILSVRC2012). Although the teacher network behaves less powerful, the students show a persistent ability growth and eventually achieve higher classification accuracies than other competitors. Model ensemble and transfer feature extraction also verify the effectiveness of our approach.