mlsanityData to 14 Oct 2026

PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation

arXiv 2103.00053 ↗First posted 26 Feb 2021cs.LG cs.CVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

One of the most efficient methods for model compression is hint distillation, where the student model is injected with information (hints) from several different layers of the teacher model. Although the selection of hint points can drastically alter the compression performance, conventional distillation approaches overlook this fact and use the same hint points as in the early studies. Therefore, we propose a clustering based hint selection methodology, where the layers of teacher model are clustered with respect to several metrics and the cluster centers are used as the hint points. Our method is applicable for any student network, once it is applied on a chosen teacher network. The proposed approach is validated in CIFAR-100 and ImageNet datasets, using various teacher-student pairs and numerous hint distillation methods. Our results show that hint points selected by our algorithm results in superior compression performance compared to state-of-the-art knowledge distillation algorithms on the same student models and datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 44 of its 79 references that are in the map. First posted on arXiv 26 Feb 2021.

Builds on

44 of 79 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching5 Feb
  2. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective1 Feb

Other references · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2016Wide Residual Networkscited by 3,070
  8. 2018YOLOv3: An Incremental Improvementcited by 2,659
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  10. 2019Searching for MobileNetV3cited by 1,994
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,653
30 more
  1. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  2. 2014How transferable are features in deep neural networks?cited by 1,225
  3. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  4. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  5. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  6. 2019Relational Knowledge Distillationcited by 889
  7. 2019Contrastive Representation Distillationcited by 748
  8. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  9. 2019Similarity-Preserving Knowledge Distillationcited by 516
  10. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  11. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 434
  12. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  13. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  14. 2019Variational Information Distillation for Knowledge Transfercited by 324
  15. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  16. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 301
  17. 2019Correlation Congruence for Knowledge Distillationcited by 285
  18. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  19. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  20. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  21. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  22. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  23. 2019Deep Learning in Video Multi-Object Tracking: A Surveycited by 107
  24. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 66
  25. 2019Learning What and Where to Transfercited by 43
  26. 2020Multi-label Contrastive Predictive Codingcited by 32
  27. 2019The State of Knowledge Distillation for Classificationcited by 9
  28. 2020Interactive Knowledge Distillationcited by 4
  29. 2020Compressing Deep Neural Networks via Layer Fusioncited by 4
  30. 2020Reducing the Teacher-Student Gap via Spherical Knowledge Disitllationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Hint-dynamic Knowledge Distillation
  2. 2022InDistill: Information flow-preserving knowledge distillation for model compression
  3. 2021Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation
  4. 2023Deep Collective Knowledge Distillation
  5. 2021Knowledge Distillation as Semiparametric Inference
  6. 2024Improve Knowledge Distillation via Label Revision and Data Selection
  7. 2021Spirit Distillation: A Model Compression Method with Multi-domain Knowledge Transfer
  8. 2021Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression