# Knowledge Distillation in Deep Learning and its Applications

arXiv: https://arxiv.org/abs/2007.09029 · HTML: https://mlsanity.com/paper/2007.09029/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2020.

## Built on by (most cited first, 29 of 29)

- [Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application](https://mlsanity.com/paper/2407.01885/) (2024)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [Knowledge Distillation in Vision Transformers: A Critical Review](https://mlsanity.com/paper/2302.02108/) (2023)
- [Applications of Knowledge Distillation in Remote Sensing: A Survey](https://mlsanity.com/paper/2409.12111/) (2024)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentation](https://mlsanity.com/paper/2307.04470/) (2023)
- [Loss Distillation via Gradient Matching for Point Cloud Completion with Weighted Chamfer Distance](https://mlsanity.com/paper/2409.06171/) (2024)
- [Low-latency Real-time Voice Conversion on CPU](https://mlsanity.com/paper/2311.00873/) (2023)
- [Recent Advances, Applications, and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2023 Symposium](https://mlsanity.com/paper/2403.01628/) (2024)
- [Larger models yield better results? Streamlined severity classification of ADHD-related concerns using BERT-based knowledge distillation](https://mlsanity.com/paper/2411.00052/) (2024)
- [Cycle Training with Semi-Supervised Domain Adaptation: Bridging Accuracy and Efficiency for Real-Time Mobile Scene Detection](https://mlsanity.com/paper/2504.09297/) (2025)
- [EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models](https://mlsanity.com/paper/2607.11012/) (2026)
- [Efficiently Robustify Pre-trained Models](https://mlsanity.com/paper/2309.07499/) (2023)
- [Weakly Supervised Change Detection via Knowledge Distillation and Multiscale Sigmoid Inference](https://mlsanity.com/paper/2403.05796/) (2024)
- [Optimizing Parking Space Classification: Distilling Ensembles into Lightweight Classifiers](https://mlsanity.com/paper/2410.14705/) (2024)
- [Practical Knowledge Distillation: Using DNNs to Beat DNNs](https://mlsanity.com/paper/2302.12360/) (2023)
- [GOFAI meets Generative AI: Development of Expert Systems by means of Large Language Models](https://mlsanity.com/paper/2507.13550/) (2025)
- [Local Feature Matching with Transformers for low-end devices](https://mlsanity.com/paper/2202.00770/) (2022)
- [Language Model Decomposition: Quantifying the Dependency and Correlation of Language Models](https://mlsanity.com/paper/2210.10289/) (2022)
- [Physics-Enhanced TinyML for Real-Time Detection of Ground Magnetic Anomalies](https://mlsanity.com/paper/2311.11452/) (2023)
- [InFiConD: Interactive No-code Fine-tuning with Concept-based Knowledge Distillation](https://mlsanity.com/paper/2406.17838/) (2024)
- [Local-to-Global Self-Supervised Representation Learning for Diabetic Retinopathy Grading](https://mlsanity.com/paper/2410.00779/) (2024)
- [Knowledge Distillation with Adapted Weight](https://mlsanity.com/paper/2501.02705/) (2025)
- [Distilling Knowledge for Designing Computational Imaging Systems](https://mlsanity.com/paper/2501.17898/) (2025)
- [Defense against Unauthorized Distillation in Image Restoration via Feature Space Perturbation](https://mlsanity.com/paper/2510.08925/) (2025)
- [Unsupervised Continual Clustering via Forward-Backward Knowledge Distillation](https://mlsanity.com/paper/2606.07474/) (2026)
- [Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it](https://mlsanity.com/paper/2608.23752/) (2026)

## Abstract (arXiv)

Deep learning based models are relatively large, and it is hard to deploy such models on resource-limited devices such as mobile phones and embedded devices. One possible solution is knowledge distillation whereby a smaller model (student model) is trained by utilizing the information from a larger model (teacher model). In this paper, we present a survey of knowledge distillation techniques applied to deep learning models. To compare the performances of different techniques, we propose a new metric called distillation metric. Distillation metric compares different knowledge distillation algorithms based on sizes and accuracy scores. Based on the survey, some interesting conclusions are drawn and presented in this paper.