mlsanityData to 7 Oct 2026

Distilling Holistic Knowledge with Graph Neural Networks

arXiv 2108.05507 ↗First posted 12 Aug 2021cs.CVcited by 26

Not on the map. Why?

Abstract (arXiv)

Knowledge Distillation (KD) aims at transferring knowledge from a larger well-optimized teacher network to a smaller learnable student network.Existing KD methods have mainly considered two types of knowledge, namely the individual knowledge and the relational knowledge. However, these two types of knowledge are usually modeled independently while the inherent correlations between them are largely ignored. It is critical for sufficient student network learning to integrate both individual knowledge and relational knowledge while reserving their inherent correlation. In this paper, we propose to distill the novel holistic knowledge based on an attributed graph constructed among instances. The holistic knowledge is represented as a unified graph-based embedding by aggregating individual knowledge from relational neighborhood samples with graph neural networks, the student network is learned by distilling the holistic knowledge in a contrastive manner. Extensive experiments and ablation studies are conducted on benchmark datasets, the results demonstrate the effectiveness of the proposed method. The code has been published in https://github.com/wyc-ruiker/HKD

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 12 Aug 2021.

Builds on

33 of 45 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2017Inductive Representation Learning on Large Graphscited by 6,284
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  11. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
  12. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
21 more
  1. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  2. 2019Relational Knowledge Distillationcited by 873
  3. 2019Contrastive Representation Distillationcited by 734
  4. 2019Similarity-Preserving Knowledge Distillationcited by 507
  5. 2013Do Deep Nets Really Need to be Deep?cited by 466
  6. 2017Few-Shot Learning with Graph Neural Networkscited by 324
  7. 2019On Mutual Information Maximization for Representation Learningcited by 315
  8. 2019Correlation Congruence for Knowledge Distillationcited by 280
  9. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  10. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  11. 2020Cross-Layer Distillation with Semantic Calibrationcited by 141
  12. 2019Online Knowledge Distillation with Diverse Peerscited by 127
  13. 2017Topology Adaptive Graph Convolutional Networkscited by 112
  14. 2018Snapshot Distillation: Teacher-Student Optimization in One Generationcited by 99
  15. 2017Multi-Label Zero-Shot Learning with Structured Knowledge Graphscited by 72
  16. 2018Graph-to-Sequence Learning using Gated Graph Neural Networkscited by 61
  17. 2018Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Studentscited by 50
  18. 2019Graph-based Knowledge Distillation by Multi-head Attention Networkcited by 31
  19. 2018Semi-supervised User Geolocation via Graph Convolutional Networkscited by 27
  20. 2019Preparing Lessons: Improve Knowledge Distillation with Better Supervisioncited by 27
  21. 2019Deep geometric knowledge distillation with graphscited by 21

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 7; 2023: 9; 2024: 6; 2025: 3.

26 papers.

Sort:

  1. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 140
  2. 2022A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directionscited by 73
  3. 2022Multimodal learning with graphscited by 54
  4. 2023Knowledge Distillation on Graphs: A Surveycited by 38
  5. 2023Visual Tuningcited by 34
  6. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  7. 2022Bending Graphs: Hierarchical Shape Matching using Gated Optimal Transportcited by 24
  8. 2023Deep Graph Reprogrammingcited by 22
  9. 2023Partition Speeds Up Learning Implicit Neural Representations Based on Exponential-Increase Hypothesiscited by 13
  10. 2023NormKD: Normalized Logits for Knowledge Distillationcited by 12
  11. 2021Graph-Based Similarity of Neural Network Representationscited by 7
  12. 2022Graph Flow: Cross-layer Graph Flow Distillation for Dual Efficient Medical Image Segmentationcited by 7
14 more
  1. 2023Graph-based Knowledge Distillation: A survey and experimental evaluationcited by 6
  2. 2022ThreshNet: An Efficient DenseNet Using Threshold Mechanism to Reduce Connectionscited by 4
  3. 2023Patch-wise Graph Contrastive Learning for Image Translationcited by 4
  4. 2024TSINR: Capturing Temporal Continuity via Implicit Neural Representations for Time Series Anomaly Detectioncited by 4
  5. 2024Focus on Focus: Focus-oriented Representation Learning and Multi-view Cross-modal Alignment for Glioma Gradingcited by 3
  6. 20243D Point Cloud Network Pruning: When Some Weights Do not Mattercited by 3
  7. 2025Optimal Brain Apoptosiscited by 3
  8. 2022Hilbert Distillation for Cross-Dimensionality Networkscited by 2
  9. 2023Knowledge-Distilled Graph Neural Networks for Personalized Epileptic Seizure Detectioncited by 1
  10. 2024Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealingcited by 0
  11. 2024Exploring Graph-based Knowledge: Multi-Level Feature Distillation via Channels Relational Graphcited by 0
  12. 2024Adaptive Teaching with Shared Classifier for Knowledge Distillationcited by 0
  13. 2025Enhancing Graph Neural Networks: A Mutual Learning Approachcited by 0
  14. 2025Logit-Based Losses Limit the Effectiveness of Feature Knowledge Distillationcited by 0