# Quantifying the Knowledge in GNNs for Reliable Distillation into MLPs

arXiv: https://arxiv.org/abs/2306.05628 · HTML: https://mlsanity.com/paper/2306.05628/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 9 Jun 2023.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Builds on (most cited first, 24 of 24)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Pitfalls of Graph Neural Network Evaluation](https://mlsanity.com/paper/1811.05868/) (2018)
- [Towards Deeper Graph Neural Networks](https://mlsanity.com/paper/2007.09296/) (2020)
- [Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation](https://mlsanity.com/paper/2110.08727/) (2021)
- [Distilling Knowledge from Graph Convolutional Networks](https://mlsanity.com/paper/2003.10477/) (2020)
- [Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework](https://mlsanity.com/paper/2103.02885/) (2021)
- [Revisiting Over-smoothing in Deep GCNs](https://mlsanity.com/paper/2003.13663/) (2020)
- [On Representation Knowledge Distillation for Graph Neural Networks](https://mlsanity.com/paper/2111.04964/) (2021)
- [GraphMixup: Improving Class-Imbalanced Node Classification on Graphs by Self-supervised Context Prediction](https://mlsanity.com/paper/2106.11133/) (2021)
- [On Self-Distilling Graph Neural Network](https://mlsanity.com/paper/2011.02255/) (2020)
- [Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework](https://mlsanity.com/paper/2305.10758/) (2023)
- [FreeKD: Free-direction Knowledge Distillation for Graph Neural Networks](https://mlsanity.com/paper/2206.06561/) (2022)
- [Deep geometric knowledge distillation with graphs](https://mlsanity.com/paper/1911.03080/) (2019)
- [Iterative Graph Self-Distillation](https://mlsanity.com/paper/2010.12609/) (2020)
- [Multi-task Self-distillation for Graph-based Semi-Supervised Learning](https://mlsanity.com/paper/2112.01174/) (2021)
- [Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classification](https://mlsanity.com/paper/2210.02097/) (2022)

## Built on by (most cited first, 25 of 25)

- [The Heterophilic Graph Learning Handbook: Benchmarks, Models, Theoretical Analysis, Applications and Challenges](https://mlsanity.com/paper/2407.09618/) (2024)
- [MAPE-PPI: Towards Effective and Efficient Protein-Protein Interaction Prediction via Microenvironment-Aware Protein Embedding](https://mlsanity.com/paper/2402.14391/) (2024)
- [Re-Dock: Towards Flexible and Realistic Molecular Docking with Diffusion Bridge](https://mlsanity.com/paper/2402.11459/) (2024)
- [Protein 3D Graph Structure Learning for Robust Structure-based Protein Property Prediction](https://mlsanity.com/paper/2310.11466/) (2023)
- [PSC-CPI: Multi-Scale Protein Sequence-Structure Contrasting for Efficient and Generalizable Compound-Protein Interaction Prediction](https://mlsanity.com/paper/2402.08198/) (2024)
- [A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation](https://mlsanity.com/paper/2403.03483/) (2024)
- [Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Tasks](https://mlsanity.com/paper/2403.01400/) (2024)
- [AdaGMLP: AdaBoosting GNN-to-MLP Knowledge Distillation](https://mlsanity.com/paper/2405.14307/) (2024)
- [LightHGNN: Distilling Hypergraph Neural Networks into MLPs for 100× Faster Inference](https://mlsanity.com/paper/2402.04296/) (2024)
- [TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillation](https://mlsanity.com/paper/2412.11180/) (2024)
- [Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigation](https://mlsanity.com/paper/2510.08962/) (2025)
- [Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillation](https://mlsanity.com/paper/2407.14768/) (2024)
- [Graph Knowledge Distillation to Mixture of Experts](https://mlsanity.com/paper/2406.11919/) (2024)
- [Teaching MLPs to Master Heterogeneous Graph-Structured Knowledge for Efficient and Accurate Inference](https://mlsanity.com/paper/2411.14035/) (2024)
- [GTAGCN: Generalized Topology Adaptive Graph Convolutional Networks](https://mlsanity.com/paper/2403.15077/) (2024)
- [ProGMLP: A Progressive Framework for GNN-to-MLP Knowledge Distillation with Efficient Trade-offs](https://mlsanity.com/paper/2507.19031/) (2025)
- [InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillation](https://mlsanity.com/paper/2601.08033/) (2026)
- [Acceleration Algorithms in GNNs: A Survey](https://mlsanity.com/paper/2405.04114/) (2024)
- [Learning to Model Graph Structural Information on MLPs via Graph Structure Self-Contrasting](https://mlsanity.com/paper/2409.05573/) (2024)
- [Sparse Decomposition of Graph Neural Networks](https://mlsanity.com/paper/2410.19723/) (2024)
- [Enhancing Transformer with GNN Structural Knowledge via Distillation: A Novel Approach](https://mlsanity.com/paper/2503.01888/) (2025)
- [Heuristic Methods are Good Teachers to Distill MLPs for Graph Link Prediction](https://mlsanity.com/paper/2504.06193/) (2025)
- [Efficient Graph Knowledge Distillation from GNNs to Kolmogorov--Arnold Networks via Self-Attention Dynamic Sampling](https://mlsanity.com/paper/2509.00560/) (2025)
- [Dual-Kernel Graph Community Contrastive Learning](https://mlsanity.com/paper/2511.08287/) (2025)
- [JDCNet: Confidence-Gated Privileged-Modality Distillation for Cost-Preserving X-ray Inference](https://mlsanity.com/paper/2603.29167/) (2026)

## Abstract (arXiv)

To bridge the gaps between topology-aware Graph Neural Networks (GNNs) and inference-efficient Multi-Layer Perceptron (MLPs), GLNN proposes to distill knowledge from a well-trained teacher GNN into a student MLP. Despite their great progress, comparatively little work has been done to explore the reliability of different knowledge points (nodes) in GNNs, especially their roles played during distillation. In this paper, we first quantify the knowledge reliability in GNN by measuring the invariance of their information entropy to noise perturbations, from which we observe that different knowledge points (1) show different distillation speeds (temporally); (2) are differentially distributed in the graph (spatially). To achieve reliable distillation, we propose an effective approach, namely Knowledge-inspired Reliable Distillation (KRD), that models the probability of each node being an informative and reliable knowledge point, based on which we sample a set of additional reliable knowledge points as supervision for training student MLPs. Extensive experiments show that KRD improves over the vanilla MLPs by 12.62% and outperforms its corresponding teacher GNNs by 2.16% averaged over 7 datasets and 3 GNN architectures.