mlsanityData to 7 Oct 2026

Quantifying the Knowledge in GNNs for Reliable Distillation into MLPs

arXiv 2306.05628 ↗First posted 9 Jun 2023cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

To bridge the gaps between topology-aware Graph Neural Networks (GNNs) and inference-efficient Multi-Layer Perceptron (MLPs), GLNN proposes to distill knowledge from a well-trained teacher GNN into a student MLP. Despite their great progress, comparatively little work has been done to explore the reliability of different knowledge points (nodes) in GNNs, especially their roles played during distillation. In this paper, we first quantify the knowledge reliability in GNN by measuring the invariance of their information entropy to noise perturbations, from which we observe that different knowledge points (1) show different distillation speeds (temporally); (2) are differentially distributed in the graph (spatially). To achieve reliable distillation, we propose an effective approach, namely Knowledge-inspired Reliable Distillation (KRD), that models the probability of each node being an informative and reliable knowledge point, based on which we sample a set of additional reliable knowledge points as supervision for training student MLPs. Extensive experiments show that KRD improves over the vanilla MLPs by 12.62% and outperforms its corresponding teacher GNNs by 2.16% averaged over 7 datasets and 3 GNN architectures.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

24 of 36 references.

References · 24

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2017Graph Attention Networkscited by 8,854
  4. 2017Inductive Representation Learning on Large Graphscited by 6,284
  5. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  6. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  7. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  8. 2020Knowledge Distillation: A Surveycited by 1,578
  9. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
  10. 2018Pitfalls of Graph Neural Network Evaluationcited by 994
  11. 2020Towards Deeper Graph Neural Networkscited by 296
  12. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
12 more
  1. 2020Distilling Knowledge from Graph Convolutional Networkscited by 128
  2. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Frameworkcited by 69
  3. 2020Revisiting Over-smoothing in Deep GCNscited by 55
  4. 2021On Representation Knowledge Distillation for Graph Neural Networkscited by 38
  5. 2021GraphMixup: Improving Class-Imbalanced Node Classification on Graphs by Self-supervised Context Predictioncited by 36
  6. 2020On Self-Distilling Graph Neural Networkcited by 35
  7. 2023Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Frameworkcited by 24
  8. 2022FreeKD: Free-direction Knowledge Distillation for Graph Neural Networkscited by 22
  9. 2019Deep geometric knowledge distillation with graphscited by 21
  10. 2020Iterative Graph Self-Distillationcited by 21
  11. 2021Multi-task Self-distillation for Graph-based Semi-Supervised Learningcited by 8
  12. 2022Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classificationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 16; 2025: 6; 2026: 2.

25 papers.

Sort:

By area:

  1. 2024The Heterophilic Graph Learning Handbook: Benchmarks, Models, Theoretical Analysis, Applications and Challengescited by 29
  2. 2024MAPE-PPI: Towards Effective and Efficient Protein-Protein Interaction Prediction via Microenvironment-Aware Protein Embeddingcited by 22
  3. 2024Re-Dock: Towards Flexible and Realistic Molecular Docking with Diffusion Bridgecited by 16
  4. 2023Protein 3D Graph Structure Learning for Robust Structure-based Protein Property Predictioncited by 13
  5. 2024PSC-CPI: Multi-Scale Protein Sequence-Structure Contrasting for Efficient and Generalizable Compound-Protein Interaction Predictioncited by 10
  6. 2024A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillationcited by 8
  7. 2024Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Taskscited by 7
  8. 2024AdaGMLP: AdaBoosting GNN-to-MLP Knowledge Distillationcited by 6
  9. 2024LightHGNN: Distilling Hypergraph Neural Networks into MLPs for 100× Faster Inferencecited by 5
  10. 2025Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigationcited by 5
  11. 2024Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillationcited by 4
  12. 2024TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillationcited by 4
13 more
  1. 2024Graph Knowledge Distillation to Mixture of Expertscited by 2
  2. 2024Teaching MLPs to Master Heterogeneous Graph-Structured Knowledge for Efficient and Accurate Inferencecited by 2
  3. 2024GTAGCN: Generalized Topology Adaptive Graph Convolutional Networkscited by 1
  4. 2025ProGMLP: A Progressive Framework for GNN-to-MLP Knowledge Distillation with Efficient Trade-offscited by 1
  5. 2026InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillationcited by 1
  6. 2024Acceleration Algorithms in GNNs: A Surveycited by 0
  7. 2024Learning to Model Graph Structural Information on MLPs via Graph Structure Self-Contrastingcited by 0
  8. 2024Sparse Decomposition of Graph Neural Networkscited by 0
  9. 2025Enhancing Transformer with GNN Structural Knowledge via Distillation: A Novel Approachcited by 0
  10. 2025Heuristic Methods are Good Teachers to Distill MLPs for Graph Link Predictioncited by 0
  11. 2025Efficient Graph Knowledge Distillation from GNNs to Kolmogorov--Arnold Networks via Self-Attention Dynamic Samplingcited by 0
  12. 2025Dual-Kernel Graph Community Contrastive Learningcited by 0
  13. 2026JDCNet: Confidence-Gated Privileged-Modality Distillation for Cost-Preserving X-ray Inferencecited by 0