mlsanityData to 7 Oct 2026

Improving Knowledge Distillation via Transferring Learning Ability

arXiv 2304.11923 ↗First posted 24 Apr 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Existing knowledge distillation methods generally use a teacher-student approach, where the student network solely learns from a well-trained teacher. However, this approach overlooks the inherent differences in learning abilities between the teacher and student networks, thus causing the capacity-gap problem. To address this limitation, we propose a novel method called SLKD.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 24 Apr 2023.

Builds on

37 of 49 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2017Mask R-CNNcited by 7,259
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2017Squeeze-and-Excitation Networkscited by 4,763
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2016Wide Residual Networkscited by 3,057
  11. 2016Pyramid Scene Parsing Networkcited by 2,472
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,619
25 more
  1. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  2. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  3. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  4. 2019Relational Knowledge Distillationcited by 873
  5. 2019Contrastive Representation Distillationcited by 734
  6. 2017Deep Mutual Learningcited by 621
  7. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  8. 2018Born Again Neural Networkscited by 539
  9. 2019Similarity-Preserving Knowledge Distillationcited by 507
  10. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  11. 2022Decoupled Knowledge Distillationcited by 380
  12. 2019On the Efficacy of Knowledge Distillationcited by 377
  13. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  14. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  15. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  16. 2019Correlation Congruence for Knowledge Distillationcited by 280
  17. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  18. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  19. 2015Unifying distillation and privileged informationcited by 218
  20. 2021Towards Understanding Knowledge Distillationcited by 158
  21. 2018Snapshot Distillation: Teacher-Student Optimization in One Generationcited by 99
  22. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 82
  23. 2020Explaining Knowledge Distillation by Quantifying the Knowledgecited by 65
  24. 2022Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classificationcited by 19
  25. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teachercited by 16

Built on by

Not cited yet.