mlsanityData to 7 Oct 2026

Improved Knowledge Distillation via Teacher Assistant

arXiv 1902.03393 ↗First posted 9 Feb 2019cs.LG cs.AI stat.MLcited by 567

Not on the map. Why?

Abstract (arXiv)

Despite the fact that deep neural networks are powerful models and achieve appealing results on many tasks, they are too large to be deployed on edge devices like smartphones or embedded sensor nodes. There have been efforts to compress these networks, and a popular method is knowledge distillation, where a large (teacher) pre-trained network is used to train a smaller (student) network. However, in this paper, we show that the student network performance degrades when the gap between student and teacher is large. Given a fixed student network, one cannot employ an arbitrarily large teacher, or in other words, a teacher can effectively transfer its knowledge to students up to a certain size, not smaller. To alleviate this shortcoming, we introduce multi-step knowledge distillation, which employs an intermediate-sized network (teacher assistant) to bridge the gap between the student and the teacher. Moreover, we study the effect of teacher assistant size and extend the framework to multi-step distillation. Theoretical analysis and extensive experiments on CIFAR-10,100 and ImageNet datasets and on CNN and ResNet architectures substantiate the effectiveness of our proposed approach.

Key facts. Cited by 567 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 72; 2022: 104; 2023: 117; 2024: 118; 2025: 82; 2026: 74.

567 papers.

Sort:

By area:

  1. 2024Depth Anything V2cited by 1,385
  2. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  3. 2022Decoupled Knowledge Distillationcited by 380
  4. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  5. 2022Knowledge Distillation from A Stronger Teachercited by 204
  6. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  7. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  8. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  9. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  10. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  11. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  12. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
38 more
  1. 2022Asymmetric Student-Teacher Networks for Industrial Anomaly Detectioncited by 115
  2. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 114
  3. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109
  4. 2021Distilling Object Detectors via Decoupled Featurescited by 105
  5. 2024Logit Standardization in Knowledge Distillationcited by 101
  6. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  7. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  8. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 84
  9. 2023Decoupled Multimodal Distilling for Emotion Recognitioncited by 76
  10. 2024NetLLM: Adapting Large Language Models for Networkingcited by 76
  11. 2024D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinementcited by 73
  12. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  13. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
  14. 2022Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generationcited by 67
  15. 2023Object-Aware Distillation Pyramid for Open-Vocabulary Object Detectioncited by 63
  16. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  17. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  18. 2023Class Attention Transfer Based Knowledge Distillationcited by 61
  19. 2023Decoupled Kullback-Leibler Divergence Losscited by 58
  20. 2021Localization Distillation for Dense Object Detectioncited by 55
  21. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  22. 2021A Survey on Green Deep Learningcited by 52
  23. 2023Knowledge Diffusion for Distillationcited by 51
  24. 2023CrossKD: Cross-Head Knowledge Distillation for Object Detectioncited by 50
  25. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  26. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
  27. 2021Multi-Target Domain Adaptation with Collaborative Consistency Learningcited by 43
  28. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 43
  29. 2021SIGN: Spatial-information Incorporated Generative Network for Generalized Zero-shot Semantic Segmentationcited by 41
  30. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  31. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  32. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 41
  33. 2024Tiny Machine Learning: Progress and Futurescited by 40
  34. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  35. 2021Learning Compatible Embeddingscited by 37
  36. 2022Efficient Knowledge Distillation from Model Checkpointscited by 37
  37. 2021Undistillable: Making A Nasty Teacher That CANNOT teach studentscited by 36
  38. 2021PocketNet: Extreme Lightweight Face Recognition Network using Neural Architecture Search and Multi-Step Knowledge Distillationcited by 36

50 most cited of 567 All 567 (JSON)