mlsanityData to 14 Oct 2026

HIRE: Distilling High-order Relational Knowledge From Heterogeneous Graph Neural Networks

arXiv 2207.11887 ↗First posted 25 Jul 2022cs.LGcited by 2

On the map: Knowledge distillation → Graph neural network distillation

Abstract (arXiv)

Researchers have recently proposed plenty of heterogeneous graph neural networks (HGNNs) due to the ubiquity of heterogeneous graphs in both academic and industrial areas. Instead of pursuing a more powerful HGNN model, in this paper, we are interested in devising a versatile plug-and-play module, which accounts for distilling relational knowledge from pre-trained HGNNs. To the best of our knowledge, we are the first to propose a HIgh-order RElational (HIRE) knowledge distillation framework on heterogeneous graphs, which can significantly boost the prediction performance regardless of model architectures of HGNNs. Concretely, our HIRE framework initially performs first-order node-level knowledge distillation, which encodes the semantics of the teacher HGNN with its prediction logits. Meanwhile, the second-order relation-level knowledge distillation imitates the relational correlation between node embeddings of different types generated by the teacher HGNN. Extensive experiments on various popular HGNNs models and three real-world heterogeneous graphs demonstrate that our method obtains consistent and considerable performance enhancement, proving its effectiveness and generalization ability.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation. Builds on 38 of its 63 references that are in the map. First posted on arXiv 25 Jul 2022.

Builds on

38 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework4 Mar

Other references · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2017Graph Attention Networkscited by 8,923
  7. 2017Inductive Representation Learning on Large Graphscited by 6,320
  8. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  9. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  11. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
25 more
  1. 2018Link Prediction Based on Graph Neural Networkscited by 832
  2. 2019Heterogeneous Graph Attention Networkcited by 734
  3. 2017Deep Mutual Learningcited by 628
  4. 2020Heterogeneous Graph Transformercited by 582
  5. 2018Born Again Neural Networkscited by 546
  6. 2018Deep Learning on Graphs: A Surveycited by 393
  7. 2020MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embeddingcited by 304
  8. 2019Correlation Congruence for Knowledge Distillationcited by 285
  9. 2019A Fair Comparison of Graph Neural Networks for Graph Classificationcited by 244
  10. 2019Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networkscited by 242
  11. 2020Machine Learning on Graphs: A Model and Comprehensive Taxonomycited by 156
  12. 2017Heterogeneous Information Network Embedding for Recommendationcited by 133
  13. 2020Distilling Knowledge from Graph Convolutional Networkscited by 129
  14. 2020A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sourcescited by 126
  15. 2019A Gentle Introduction to Deep Learning for Graphscited by 108
  16. 2019EEG-Based Emotion Recognition Using Regularized Graph Neural Networkscited by 97
  17. 2020Heterogeneous Graph Neural Networks for Malicious Account Detectioncited by 96
  18. 2021Graph Neural Networks in Network Neurosciencecited by 92
  19. 2020Molecular graph generation with Graph Neural Networkscited by 80
  20. 2020On Self-Distilling Graph Neural Networkcited by 35
  21. 2021BrainNNExplainer: An Interpretable Graph Neural Network Framework for Brain Network based Disease Analysiscited by 14
  22. 2019Multi-scale Anomaly Detection on Attributed Networkscited by 13
  23. 2021KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillationcited by 11
  24. 2020Hybrid Micro/Macro Level Convolution for Heterogeneous Graph Learningcited by 9
  25. 2019Binarized Collaborative Filtering with Distilling Graph Convolutional Networkscited by 8

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023Graph-based Knowledge Distillation: A survey and experimental evaluationcited by 6
  2. 2024Teaching MLPs to Master Heterogeneous Graph-Structured Knowledge for Efficient and Accurate Inferencecited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023Knowledge Distillation on Graphs: A Survey
  2. 2022Weisfeiler and Leman Go Relational
  3. 2024A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation
  4. 2021Distilling Holistic Knowledge with Graph Neural Networks
  5. 2022Linkless Link Prediction via Relational Distillation
  6. 2022Boosting Graph Neural Networks via Adaptive Knowledge Distillation
  7. 2026InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillation
  8. 2022Alignahead: Online Cross-Layer Knowledge Extraction on Graph Neural Networks