mlsanityData to 7 Oct 2026

Knowledge Distillation in Deep Learning and its Applications

arXiv 2007.09029 ↗First posted 17 Jul 2020cs.LG stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

Deep learning based models are relatively large, and it is hard to deploy such models on resource-limited devices such as mobile phones and embedded devices. One possible solution is knowledge distillation whereby a smaller model (student model) is trained by utilizing the information from a larger model (teacher model). In this paper, we present a survey of knowledge distillation techniques applied to deep learning models. To compare the performances of different techniques, we propose a new metric called distillation metric. Distillation metric compares different knowledge distillation algorithms based on sizes and accuracy scores. Based on the survey, some interesting conclusions are drawn and presented in this paper.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 7; 2024: 9; 2025: 6; 2026: 3.

29 papers.

Sort:

  1. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  2. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  3. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  4. 2022Teacher-Student Architecture for Knowledge Learning: A Surveycited by 23
  5. 2023Knowledge Distillation in Vision Transformers: A Critical Reviewcited by 13
  6. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  7. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  8. 2023Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentationcited by 4
  9. 2024Loss Distillation via Gradient Matching for Point Cloud Completion with Weighted Chamfer Distancecited by 4
  10. 2023Low-latency Real-time Voice Conversion on CPUcited by 3
  11. 2024Recent Advances, Applications, and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2023 Symposiumcited by 3
  12. 2024Larger models yield better results? Streamlined severity classification of ADHD-related concerns using BERT-based knowledge distillationcited by 3
17 more
  1. 2025Cycle Training with Semi-Supervised Domain Adaptation: Bridging Accuracy and Efficiency for Real-Time Mobile Scene Detectioncited by 3
  2. 2026EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Modelscited by 3
  3. 2023Efficiently Robustify Pre-trained Modelscited by 2
  4. 2024Weakly Supervised Change Detection via Knowledge Distillation and Multiscale Sigmoid Inferencecited by 2
  5. 2024Optimizing Parking Space Classification: Distilling Ensembles into Lightweight Classifierscited by 2
  6. 2023Practical Knowledge Distillation: Using DNNs to Beat DNNscited by 1
  7. 2025GOFAI meets Generative AI: Development of Expert Systems by means of Large Language Modelscited by 1
  8. 2022Local Feature Matching with Transformers for low-end devicescited by 0
  9. 2022Language Model Decomposition: Quantifying the Dependency and Correlation of Language Modelscited by 0
  10. 2023Physics-Enhanced TinyML for Real-Time Detection of Ground Magnetic Anomaliescited by 0
  11. 2024InFiConD: Interactive No-code Fine-tuning with Concept-based Knowledge Distillationcited by 0
  12. 2024Local-to-Global Self-Supervised Representation Learning for Diabetic Retinopathy Gradingcited by 0
  13. 2025Knowledge Distillation with Adapted Weightcited by 0
  14. 2025Distilling Knowledge for Designing Computational Imaging Systemscited by 0
  15. 2025Defense against Unauthorized Distillation in Image Restoration via Feature Space Perturbationcited by 0
  16. 2026Unsupervised Continual Clustering via Forward-Backward Knowledge Distillationcited by 0
  17. 2026Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid itcited by 0