mlsanityData to 7 Oct 2026

Interpretable Embedding Procedure Knowledge Transfer via Stacked Principal Component Analysis and Graph Neural Network

arXiv 2104.13561 ↗First posted 28 Apr 2021cs.CV cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation (KD) is one of the most useful techniques for light-weight neural networks. Although neural networks have a clear purpose of embedding datasets into the low-dimensional space, the existing knowledge was quite far from this purpose and provided only limited information. We argue that good knowledge should be able to interpret the embedding procedure. This paper proposes a method of generating interpretable embedding procedure (IEP) knowledge based on principal component analysis, and distilling it based on a message passing neural network. Experimental results show that the student network trained by the proposed KD method improves 2.28% in the CIFAR100 dataset, which is higher performance than the state-of-the-art (SOTA) method. We also demonstrate that the embedding procedure knowledge is interpretable via visualization of the proposed KD process. The implemented code is available at https://github.com/sseung0703/IEPKT.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 24 of its 40 references that are in the map. First posted on arXiv 28 Apr 2021.

Builds on

24 of 40 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  8. 2016Wide Residual Networkscited by 3,057
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  10. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  11. 2019Relational Knowledge Distillationcited by 873
  12. 2016Language Modeling with Gated Convolutional Networkscited by 688
12 more
  1. 2017Deep Mutual Learningcited by 621
  2. 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
  3. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  4. 2019Variational Information Distillation for Knowledge Transfercited by 321
  5. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  6. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  7. 2017End-to-end representation learning for Correlation Filter based trackingcited by 90
  8. 2018Linguistically-Informed Self-Attention for Semantic Role Labelingcited by 71
  9. 2018Low-resolution Face Recognition in the Wild via Selective Knowledge Distillationcited by 55
  10. 2019Learning Symbolic Physics with Graph Networkscited by 52
  11. 2018Self-supervised Knowledge Distillation Using Singular Value Decompositioncited by 47
  12. 2017Semi-supervised learning of hierarchical representations of molecules using neural message passingcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2.

3 papers.

  1. 2023Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalizationcited by 28
  2. 2022PCA-Based Knowledge Distillation Towards Lightweight and Content-Style Balanced Photorealistic Style Transfer Modelscited by 11
  3. 2023Graph-based Knowledge Distillation: A survey and experimental evaluationcited by 6