mlsanityData to 7 Oct 2026

Knowledge Distillation: A Survey

arXiv 2006.05525 ↗First posted 9 Jun 2020cs.LG stat.MLcited by 1,578

Not on the map. Why?

Cited by 33 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements. To this end, a variety of model compression and acceleration techniques have been developed. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model. It has received rapid increasing attention from the community. This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher-student architecture, distillation algorithms, performance comparison and applications. Furthermore, challenges in knowledge distillation are briefly reviewed and comments on future research are discussed and forwarded.

Key facts. Cited by 1,578 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 94; 2022: 239; 2023: 327; 2024: 346; 2025: 361; 2026: 211.

1,578 papers.

Sort:

By area:

  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  4. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  5. 2021ImageNet-21K Pretraining for the Massescited by 607
  6. 2024Large Language Models: A Surveycited by 506
  7. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  8. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  9. 2021Deep Long-Tailed Learning: A Surveycited by 416
  10. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  11. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  12. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
38 more
  1. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  2. 2022Large Language Models Are Reasoning Teacherscited by 318
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  4. 2023A Survey on Model Compression for Large Language Modelscited by 263
  5. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  6. 2022A Survey on Generative Diffusion Modelcited by 238
  7. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  8. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 216
  9. 2022Knowledge Distillation from A Stronger Teachercited by 204
  10. 2022Medical Image Segmentation Review: The success of U-Netcited by 190
  11. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  12. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  13. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 140
  14. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  15. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  16. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  17. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  18. 2022Dataset Distillation via Factorizationcited by 115
  19. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  20. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  21. 2021General Instance Distillation for Object Detectioncited by 113
  22. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109
  23. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  24. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 96
  25. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
  26. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  27. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  28. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  29. 2023DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultationcited by 89
  30. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  31. 2023Cascade Speculative Drafting for Even Faster LLM Inferencecited by 83
  32. 2024TSD-SR: One-Step Diffusion with Target Score Distillation for Real-World Image Super-Resolutioncited by 83
  33. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  34. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  35. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  36. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  37. 2022Language Models Can Teach Themselves to Program Bettercited by 75
  38. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73

50 most cited of 1,578 All 1,578 (JSON)