mlsanityData to 7 Oct 2026

On the Efficacy of Knowledge Distillation

arXiv 1910.01348 ↗First posted 3 Oct 2019cs.LG cs.CVcited by 377

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a thorough evaluation of the efficacy of knowledge distillation and its dependence on student and teacher architectures. Starting with the observation that more accurate teachers often don't make good teachers, we attempt to tease apart the factors that affect knowledge distillation performance. We find crucially that larger models do not often make better teachers. We show that this is a consequence of mismatched capacity, and that small students are unable to mimic large teachers. We find typical ways of circumventing this (such as performing a sequence of knowledge distillation steps) to be ineffective. Finally, we show that this effect can be mitigated by stopping the teacher's training early. Our results generalize across datasets and models.

Key facts. Cited by 377 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 53; 2022: 72; 2023: 74; 2024: 73; 2025: 51; 2026: 54.

377 papers.

Sort:

By area:

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  2. 2022Decoupled Knowledge Distillationcited by 380
  3. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  4. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  5. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  6. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  7. 2022Knowledge Distillation from A Stronger Teachercited by 204
  8. 2021Does Knowledge Distillation Really Work?cited by 187
  9. 2022FlexiViT: One Model for All Patch Sizescited by 151
  10. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  11. 2023Getting aligned on representational alignmentcited by 142
  12. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
38 more
  1. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  2. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109
  3. 2022Masked Generative Distillationcited by 107
  4. 2024Logit Standardization in Knowledge Distillationcited by 101
  5. 2022Expanding Small-Scale Datasets with Guided Imaginationcited by 61
  6. 2023Class Attention Transfer Based Knowledge Distillationcited by 61
  7. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 60
  8. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 58
  9. 2023Decoupled Kullback-Leibler Divergence Losscited by 58
  10. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  11. 2024From LLM to NMT: Advancing Low-Resource Machine Translation with Claudecited by 57
  12. 2022MAViL: Masked Audio-Video Learnerscited by 56
  13. 2021DeepReDuce: ReLU Reduction for Fast Private Inferencecited by 50
  14. 2023CrossKD: Cross-Head Knowledge Distillation for Object Detectioncited by 50
  15. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  16. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillationcited by 46
  17. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 45
  18. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  19. 2022Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillationcited by 44
  20. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
  21. 2021MetaCorrection: Domain-aware Meta Loss Correction for Unsupervised Domain Adaptation in Semantic Segmentationcited by 43
  22. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  23. 2023Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillationcited by 41
  24. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 41
  25. 2021Co-advise: Cross Inductive Bias Distillationcited by 40
  26. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  27. 2022Efficient Knowledge Distillation from Model Checkpointscited by 37
  28. 2022Masked Autoencoders Enable Efficient Knowledge Distillerscited by 36
  29. 2021Meta-Teacher For Face Anti-Spoofingcited by 33
  30. 2024Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 32
  31. 2021Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERTcited by 31
  32. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  33. 2022A Closer Look at Self-Supervised Lightweight Vision Transformerscited by 31
  34. 2021Knowledge Distillation as Semiparametric Inferencecited by 29
  35. 2022Switchable Online Knowledge Distillationcited by 29
  36. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  37. 2023Multimodal Distillation for Egocentric Action Recognitioncited by 28
  38. 2021Diversity Matters When Learning From Ensemblescited by 27

50 most cited of 377 All 377 (JSON)