# A Comprehensive Survey on Knowledge Distillation

arXiv: https://arxiv.org/abs/2503.12067 · HTML: https://mlsanity.com/paper/2503.12067/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- First posted on arXiv 15 Mar 2025.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Built on by (most cited first, 30 of 51)

- [Q-KVComm: Efficient Multi-Agent Communication Via Adaptive KV Cache Compression](https://mlsanity.com/paper/2512.17914/) (2025)
- [LaSER: Internalizing Explicit Reasoning into Latent Space for Dense Retrieval](https://mlsanity.com/paper/2603.01425/) (2026)
- [From Performance to Practice: Knowledge-Distilled Segmentator for On-Premises Clinical Workflows](https://mlsanity.com/paper/2601.09191/) (2026)
- [Sustained Gradient Alignment Mediates Subliminal Learning in a Multi-Step Setting: Evidence from MNIST Auxiliary Logit Distillation Experiment](https://mlsanity.com/paper/2604.25779/) (2026)
- [Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack](https://mlsanity.com/paper/2509.23871/) (2025)
- [Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation](https://mlsanity.com/paper/2608.09826/) (2026)
- [InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer](https://mlsanity.com/paper/2511.15967/) (2025)
- [Quantifying Subliminal Behavioral Transfer Ratios in Language Model Distillation](https://mlsanity.com/paper/2606.11270/) (2026)
- [Improving Knowledge Distillation Under Unknown Covariate Shift Through Confidence-Guided Data Augmentation](https://mlsanity.com/paper/2506.02294/) (2025)
- [Distilling Lightweight Domain Experts from Large ML Models by Identifying Relevant Subspaces](https://mlsanity.com/paper/2601.05913/) (2026)
- [Recursive Meta-Distillation: An Axiomatic Framework for Iterative Knowledge Refinement](https://mlsanity.com/paper/2601.13100/) (2026)
- [Privacy-Aware Continual Self-Supervised Learning on Multi-Window Chest Computed Tomography for Domain-Shift Robustness](https://mlsanity.com/paper/2510.27213/) (2025)
- [Divide, Cache, Conquer: Dichotomic Prompting for Efficient Multi-Label LLM-Based Classification](https://mlsanity.com/paper/2511.03830/) (2025)
- [LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection](https://mlsanity.com/paper/2512.05663/) (2025)
- [Adaptive Weighting in Knowledge Distillation: An Axiomatic Framework for Multi-Scale Teacher Ensemble Optimization](https://mlsanity.com/paper/2601.17910/) (2026)
- [DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision Language Models](https://mlsanity.com/paper/2605.26038/) (2026)
- [Understanding Layer Patching in Model Size Interpolation](https://mlsanity.com/paper/2607.08170/) (2026)
- [CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework](https://mlsanity.com/paper/2508.04816/) (2025)
- [DISPATCH: Distilling Selective Patches for Speech Enhancement](https://mlsanity.com/paper/2509.15922/) (2025)
- [Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning](https://mlsanity.com/paper/2511.00655/) (2025)
- [ODE-ViT: Plug & Play Attention Layer from the Generalization of the ViT as an Ordinary Differential Equation](https://mlsanity.com/paper/2511.16501/) (2025)
- [SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines](https://mlsanity.com/paper/2601.01484/) (2026)
- [ADMEDTAGGER: an annotation framework for distillation of expert knowledge for the Polish medical language](https://mlsanity.com/paper/2601.09722/) (2025)
- [Integrating Knowledge Distillation Methods: A Sequential Multi-Stage Framework](https://mlsanity.com/paper/2601.15657/) (2026)
- [Domain Specific Specialization in Low-Resource Settings: The Efficacy of Offline Response-Based Knowledge Distillation in Large Language Models](https://mlsanity.com/paper/2601.16219/) (2026)
- [Drive-KD: Multi-Teacher Distillation for VLMs in Autonomous Driving](https://mlsanity.com/paper/2601.21288/) (2026)
- [Responsible AI in Business](https://mlsanity.com/paper/2602.13244/) (2026)
- [Logit Distance Bounds Representational Similarity](https://mlsanity.com/paper/2602.15438/) (2026)
- [Enhancing Automatic Chord Recognition via Pseudo-Labeling and Knowledge Distillation](https://mlsanity.com/paper/2602.19778/) (2026)
- [Meta-Learning and Meta-Reinforcement Learning -- Tracing the Path towards DeepMind's Adaptive Agent](https://mlsanity.com/paper/2602.19837/) (2026)

## Abstract (arXiv)

Deep Neural Networks (DNNs) have achieved notable performance in the fields of computer vision and natural language processing with various applications in both academia and industry. However, with recent advancements in DNNs and transformer models with a tremendous number of parameters, deploying these large models on edge devices causes serious issues such as high runtime and memory consumption. This is especially concerning with the recent large-scale foundation models, Vision-Language Models (VLMs), and Large Language Models (LLMs). Knowledge Distillation (KD) is one of the prominent techniques proposed to address the aforementioned problems using a teacher-student architecture. More specifically, a lightweight student model is trained using additional knowledge from a cumbersome teacher model. In this work, a comprehensive survey of knowledge distillation methods is proposed. This includes reviewing KD from different aspects: distillation sources, distillation schemes, distillation algorithms, distillation by modalities, applications of distillation, and comparison among existing methods. In contrast to most existing surveys, which are either outdated or simply update former surveys, this work proposes a comprehensive survey with a new point of view and representation structure that categorizes and investigates the most recent methods in knowledge distillation. This survey considers various critically important subcategories, including KD for diffusion models, 3D inputs, foundational models, transformers, and LLMs. Furthermore, existing challenges in KD and possible future research directions are discussed. Github page of the project: https://github.com/IPL-Sharif/KD_Survey