mlsanityData to 7 Oct 2026

Towards a Unified View of Affinity-Based Knowledge Distillation

arXiv 2209.15555 ↗First posted 30 Sep 2022cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Knowledge transfer between artificial neural networks has become an important topic in deep learning. Among the open questions are what kind of knowledge needs to be preserved for the transfer, and how it can be effectively achieved. Several recent work have shown good performance of distillation methods using relation-based knowledge. These algorithms are extremely attractive in that they are based on simple inter-sample similarities. Nevertheless, a proper metric of affinity and use of it in this context is far from well understood. In this paper, by explicitly modularising knowledge distillation into a framework of three components, i.e. affinity, normalisation, and loss, we give a unified treatment of these algorithms as well as study a number of unexplored combinations of the modules. With this framework we perform extensive evaluations of numerous distillation objectives for image classification, and obtain a few useful insights for effective design choices while demonstrating how relation-based knowledge distillation could achieve comparable performance to the state of the art in spite of the simplicity.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 35 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

26 of 35 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  4. 2016Wide Residual Networkscited by 3,057
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  7. 2020Knowledge Distillation: A Surveycited by 1,578
  8. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  9. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  10. 2019Relational Knowledge Distillationcited by 873
  11. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  12. 2019Contrastive Representation Distillationcited by 734
14 more
  1. 2019Similarity-Preserving Knowledge Distillationcited by 507
  2. 2013Do Deep Nets Really Need to be Deep?cited by 466
  3. 2019Variational Information Distillation for Knowledge Transfercited by 321
  4. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  5. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  6. 2019Correlation Congruence for Knowledge Distillationcited by 280
  7. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  8. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  9. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  10. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 141
  11. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 76
  12. 2020Wasserstein Contrastive Representation Distillationcited by 63
  13. 2021Complementary Relation Contrastive Distillationcited by 42
  14. 2017Binarized Convolutional Landmark Localizers for Human Pose Estimation and Face Alignment with Limited Resourcescited by 23

Built on by

Not cited yet.