# Preparing Lessons: Improve Knowledge Distillation with Better Supervision

arXiv: https://arxiv.org/abs/1911.07471 · HTML: https://mlsanity.com/paper/1911.07471/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2019.

## Built on by (most cited first, 27 of 27)

- [Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective](https://mlsanity.com/paper/2102.00650/) (2021)
- [There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledge](https://mlsanity.com/paper/2103.01353/) (2021)
- [Distilling Holistic Knowledge with Graph Neural Networks](https://mlsanity.com/paper/2108.05507/) (2021)
- [Cross-Task Knowledge Distillation in Multi-Task Recommendation](https://mlsanity.com/paper/2202.09852/) (2022)
- [Function-Consistent Feature Distillation](https://mlsanity.com/paper/2304.11832/) (2023)
- [Distilling Efficient Vision Transformers from CNNs for Semantic Segmentation](https://mlsanity.com/paper/2310.07265/) (2023)
- [A Review of Pseudo-Labeling for Computer Vision](https://mlsanity.com/paper/2408.07221/) (2024)
- [Federated Distillation: A Survey](https://mlsanity.com/paper/2404.08564/) (2024)
- [Knowledge Distillation with Refined Logits](https://mlsanity.com/paper/2408.07703/) (2024)
- [Improve Knowledge Distillation via Label Revision and Data Selection](https://mlsanity.com/paper/2404.03693/) (2024)
- [Efficient Deep Learning Infrastructures for Embedded Computing Systems: A Comprehensive Survey and Future Envision](https://mlsanity.com/paper/2411.01431/) (2024)
- [Isotonic Data Augmentation for Knowledge Distillation](https://mlsanity.com/paper/2107.01412/) (2021)
- [KDCTime: Knowledge Distillation with Calibration on InceptionTime for Time-series Classification](https://mlsanity.com/paper/2112.02291/) (2021)
- [A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques](https://mlsanity.com/paper/2302.08571/) (2023)
- [Leveraging Angular Distributions for Improved Knowledge Distillation](https://mlsanity.com/paper/2302.14130/) (2023)
- [Enhancing Logits Distillation with Plug&Play Kendall's τ Ranking Loss](https://mlsanity.com/paper/2409.17823/) (2024)
- [Using Knowledge Distillation to improve interpretable models in a retail banking context](https://mlsanity.com/paper/2209.15496/) (2022)
- [Class-aware Information for Logit-based Knowledge Distillation](https://mlsanity.com/paper/2211.14773/) (2022)
- [Triplet Knowledge Distillation](https://mlsanity.com/paper/2305.15975/) (2023)
- [Promoting Generalized Cross-lingual Question Answering in Few-resource Scenarios via Self-knowledge Distillation](https://mlsanity.com/paper/2309.17134/) (2023)
- [PCSR: Pseudo-label Consistency-Guided Sample Refinement for Noisy Correspondence Learning](https://mlsanity.com/paper/2509.15623/) (2025)
- [GaitKD: A Universal Decoupled Distillation Framework for Efficient Gait Recognition](https://mlsanity.com/paper/2604.26255/) (2026)
- [Exploring Graph-based Knowledge: Multi-Level Feature Distillation via Channels Relational Graph](https://mlsanity.com/paper/2405.08547/) (2024)
- [Intra-class Patch Swap for Self-Distillation](https://mlsanity.com/paper/2505.14124/) (2025)
- [Progressive Class-level Distillation](https://mlsanity.com/paper/2505.24310/) (2025)
- [Parameter-Free Logit Distillation via Sorting Mechanism](https://mlsanity.com/paper/2508.16544/) (2025)
- [REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiency](https://mlsanity.com/paper/2602.04677/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD) is widely used for training a compact model with the supervision of another large model, which could effectively improve the performance. Previous methods mainly focus on two aspects: 1) training the student to mimic representation space of the teacher; 2) training the model progressively or adding extra module like discriminator. Knowledge from teacher is useful, but it is still not exactly right compared with ground truth. Besides, overly uncertain supervision also influences the result. We introduce two novel approaches, Knowledge Adjustment (KA) and Dynamic Temperature Distillation (DTD), to penalize bad supervision and improve student model. Experiments on CIFAR-100, CINIC-10 and Tiny ImageNet show that our methods get encouraging performance compared with state-of-the-art methods. When combined with other KD-based methods, the performance will be further improved.