mlsanityData to 14 Oct 2026

Teaching MLPs to Master Heterogeneous Graph-Structured Knowledge for Efficient and Accurate Inference

arXiv 2411.14035 ↗First posted 21 Nov 2024cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Heterogeneous Graph Neural Networks (HGNNs) have achieved promising results in various heterogeneous graph learning tasks, owing to their superiority in capturing the intricate relationships and diverse relational semantics inherent in heterogeneous graph structures. However, the neighborhood-fetching latency incurred by structure dependency in HGNNs makes it challenging to deploy for latency-constrained applications that require fast inference. Inspired by recent GNN-to-MLP knowledge distillation frameworks, we introduce HG2M and HG2M+ to combine both HGNN's superior performance and MLP's efficient inference. HG2M directly trains student MLPs with node features as input and soft labels from teacher HGNNs as targets, and HG2M+ further distills reliable and heterogeneous semantic knowledge into student MLPs through reliable node distillation and reliable meta-path distillation. Experiments conducted on six heterogeneous graph datasets show that despite lacking structural dependencies, HG2Ms can still achieve competitive or even better performance than HGNNs and significantly outperform vanilla MLPs. Moreover, HG2Ms demonstrate a 379.24× speedup in inference over HGNNs on the large-scale IGB-3M-19 dataset, showcasing their ability for latency-sensitive deployments.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 21 Nov 2024.

Builds on

33 of 51 references.

References · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2017Graph Attention Networkscited by 8,923
  7. 2017Inductive Representation Learning on Large Graphscited by 6,320
  8. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  9. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  10. 2019Heterogeneous Graph Attention Networkcited by 734
  11. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 329
  12. 2020MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embeddingcited by 304
21 more
  1. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  2. 2021Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networkscited by 176
  3. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 152
  4. 2017Heterogeneous Information Network Embedding for Recommendationcited by 133
  5. 2021Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learningcited by 130
  6. 2019Relational Graph Attention Networkscited by 82
  7. 2020Interpretable and Efficient Heterogeneous Graph Convolutional Networkcited by 48
  8. 2022Linkless Link Prediction via Relational Distillationcited by 39
  9. 2023VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPscited by 35
  10. 2020On Graph Neural Networks versus Graph-Augmented MLPscited by 30
  11. 2023IGB: Addressing The Gaps In Labeling, Features, Heterogeneity, and Size of Public Graph Datasets for Deep Learning Researchcited by 30
  12. 2019Rethinking Softmax with Cross-Entropy: Neural Network Classifier as Mutual Information Estimatorcited by 25
  13. 2023Quantifying the Knowledge in GNNs for Reliable Distillation into MLPscited by 25
  14. 2023Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Frameworkcited by 24
  15. 2024Reliable Node Similarity Matrix Guided Contrastive Graph Clusteringcited by 12
  16. 2023Deep Insights into Noisy Pseudo Labeling on Graph Datacited by 11
  17. 2019Joint embedding of structure and features via graph convolutional networkscited by 8
  18. 2024A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillationcited by 8
  19. 2024LightHGNN: Distilling Hypergraph Neural Networks into MLPs for 100× Faster Inferencecited by 5
  20. 2024MuGSI: Distilling GNNs with Multi-Granularity Structural Information for Graph Classificationcited by 5
  21. 2022HIRE: Distilling High-order Relational Knowledge From Heterogeneous Graph Neural Networkscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Multi-Scale Heterogeneous Text-Attributed Graph Datasets From Diverse Domainscited by 5
  2. 2025Relational Database Distillation: From Structured Tables to Condensed Graph Datacited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP
  2. 2023Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs
  3. 2026InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillation
  4. 2023Unveiling the Unseen Potential of Graph Learning through MLPs: Effective Graph Learners Using Propagation-Embracing MLPs
  5. 2025Learning Accurate, Efficient, and Interpretable MLPs on Multiplex Graphs via Node-wise Multi-View Ensemble Distillation
  6. 2025ProGMLP: A Progressive Framework for GNN-to-MLP Knowledge Distillation with Efficient Trade-offs
  7. 2024Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework
  8. 2023Propagate & Distill: Towards Effective Graph Learners Using Propagation-Embracing MLPs