# HIRE: Distilling High-order Relational Knowledge From Heterogeneous Graph Neural Networks

arXiv: https://arxiv.org/abs/2207.11887 · HTML: https://mlsanity.com/paper/2207.11887/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- Builds on 38 of its 63 references that are in the map.
- First posted on arXiv 25 Jul 2022.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Link Prediction Based on Graph Neural Networks](https://mlsanity.com/paper/1802.09691/) (2018)
- [Heterogeneous Graph Attention Network](https://mlsanity.com/paper/1903.07293/) (2019)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Deep Learning on Graphs: A Survey](https://mlsanity.com/paper/1812.04202/) (2018)
- [MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding](https://mlsanity.com/paper/2002.01680/) (2020)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [A Fair Comparison of Graph Neural Networks for Graph Classification](https://mlsanity.com/paper/1912.09893/) (2019)
- [Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks](https://mlsanity.com/paper/1909.01315/) (2019)
- [Machine Learning on Graphs: A Model and Comprehensive Taxonomy](https://mlsanity.com/paper/2005.03675/) (2020)
- [Heterogeneous Information Network Embedding for Recommendation](https://mlsanity.com/paper/1711.10730/) (2017)
- [Distilling Knowledge from Graph Convolutional Networks](https://mlsanity.com/paper/2003.10477/) (2020)
- [A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sources](https://mlsanity.com/paper/2011.14867/) (2020)
- [A Gentle Introduction to Deep Learning for Graphs](https://mlsanity.com/paper/1912.12693/) (2019)
- [EEG-Based Emotion Recognition Using Regularized Graph Neural Networks](https://mlsanity.com/paper/1907.07835/) (2019)
- [Heterogeneous Graph Neural Networks for Malicious Account Detection](https://mlsanity.com/paper/2002.12307/) (2020)
- [Graph Neural Networks in Network Neuroscience](https://mlsanity.com/paper/2106.03535/) (2021)

## Built on by (most cited first, 2 of 2)

- [Graph-based Knowledge Distillation: A survey and experimental evaluation](https://mlsanity.com/paper/2302.14643/) (2023)
- [Teaching MLPs to Master Heterogeneous Graph-Structured Knowledge for Efficient and Accurate Inference](https://mlsanity.com/paper/2411.14035/) (2024)

## Abstract (arXiv)

Researchers have recently proposed plenty of heterogeneous graph neural networks (HGNNs) due to the ubiquity of heterogeneous graphs in both academic and industrial areas. Instead of pursuing a more powerful HGNN model, in this paper, we are interested in devising a versatile plug-and-play module, which accounts for distilling relational knowledge from pre-trained HGNNs. To the best of our knowledge, we are the first to propose a HIgh-order RElational (HIRE) knowledge distillation framework on heterogeneous graphs, which can significantly boost the prediction performance regardless of model architectures of HGNNs. Concretely, our HIRE framework initially performs first-order node-level knowledge distillation, which encodes the semantics of the teacher HGNN with its prediction logits. Meanwhile, the second-order relation-level knowledge distillation imitates the relational correlation between node embeddings of different types generated by the teacher HGNN. Extensive experiments on various popular HGNNs models and three real-world heterogeneous graphs demonstrate that our method obtains consistent and considerable performance enhancement, proving its effectiveness and generalization ability.