mlsanityData to 13 Oct 2026

Visualizing the embedding space to explain the effect of knowledge distillation

arXiv 2110.04483 ↗First posted 9 Oct 2021cs.CV cs.LGcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Recent research has found that knowledge distillation can be effective in reducing the size of a network and in increasing generalization. A pre-trained, large teacher network, for example, was shown to be able to bootstrap a student model that eventually outperforms the teacher in a limited label environment. Despite these advances, it still is relatively unclear why this method works, that is, what the resulting student model does 'better'. To address this issue, here, we utilize two non-linear, low-dimensional embedding methods (t-SNE and IVIS) to visualize representation spaces of different layers in a network. We perform a set of extensive experiments with different architecture parameters and distillation methods. The resulting visualizations and metrics clearly show that distillation guides the network to find a more compact representation space for higher accuracy already in earlier layers compared to its non-distilled version.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 27 of its 37 references that are in the map. First posted on arXiv 9 Oct 2021.

Builds on

27 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Towards Understanding Knowledge Distillation27 May
  2. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillation6 Mar

Other references · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,321
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,476
  4. 2016Wide Residual Networkscited by 3,063
  5. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,838
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,315
  7. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,082
  8. 2020Knowledge Distillation: A Surveycited by 1,590
  9. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  10. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,378
  11. 2019A Comprehensive Survey on Transfer Learningcited by 1,157
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,067
13 more
  1. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 471
  2. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 313
  3. 2018Model compression via distillation and quantizationcited by 287
  4. 2015Semi-Supervised Learning with Ladder Networkscited by 247
  5. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  6. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 152
  7. 2016An Analysis of Deep Neural Network Models for Practical Applicationscited by 116
  8. 2020Explaining Knowledge Distillation by Quantifying the Knowledgecited by 65
  9. 2018Clustering with Deep Learning: Taxonomy and New Methodscited by 45
  10. 2018t-SNE-CUDA: GPU-Accelerated t-SNE and its Applications to Modern Datacited by 19
  11. 2014Visualizing and Comparing Convolutional Neural Networkscited by 4
  12. 2019Distill-to-Label: Weakly Supervised Instance Labeling Using Knowledge Distillationcited by 3
  13. 2019Progressive Label Distillation: Learning Input-Efficient Deep Neural Networkscited by 1

Built on by

Not cited yet.