# Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs

arXiv: https://arxiv.org/abs/2303.13763 · HTML: https://mlsanity.com/paper/2303.13763/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 24 Mar 2023.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Builds on (most cited first, 25 of 25)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Pitfalls of Graph Neural Network Evaluation](https://mlsanity.com/paper/1811.05868/) (2018)
- [Predict then Propagate: Graph Neural Networks meet Personalized PageRank](https://mlsanity.com/paper/1810.05997/) (2018)
- [GNNExplainer: Generating Explanations for Graph Neural Networks](https://mlsanity.com/paper/1903.03894/) (2019)
- [Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View](https://mlsanity.com/paper/1909.03211/) (2019)
- [Few-Shot Learning with Graph Neural Networks](https://mlsanity.com/paper/1711.04043/) (2017)
- [Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods](https://mlsanity.com/paper/2110.14446/) (2021)
- [Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation](https://mlsanity.com/paper/2110.08727/) (2021)
- [Interpreting and Unifying Graph Neural Networks with An Optimization Framework](https://mlsanity.com/paper/2101.11859/) (2021)
- [A Unified View on Graph Neural Networks as Graph Signal Denoising](https://mlsanity.com/paper/2010.01777/) (2020)
- [Graph Meta Learning via Local Subgraphs](https://mlsanity.com/paper/2006.07889/) (2020)
- [Graph-MLP: Node Classification without Message Passing in Graph](https://mlsanity.com/paper/2106.04051/) (2021)
- [Graph Few-shot Learning via Knowledge Transfer](https://mlsanity.com/paper/1910.03053/) (2019)
- [Federated Graph Neural Networks: Overview, Techniques and Challenges](https://mlsanity.com/paper/2202.07256/) (2022)
- [Zero-shot Node Classification with Decomposed Graph Prototype Network](https://mlsanity.com/paper/2106.08022/) (2021)
- [NOSMOG: Learning Noise-robust and Structure-aware MLPs on Graphs](https://mlsanity.com/paper/2208.10010/) (2022)
- [SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP](https://mlsanity.com/paper/2210.09609/) (2022)
- [Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classification](https://mlsanity.com/paper/2210.02097/) (2022)
- [PAGE: Prototype-Based Model-Level Explanations for Graph Neural Networks](https://mlsanity.com/paper/2210.17159/) (2022)

## Built on by (most cited first, 7 of 7)

- [Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework](https://mlsanity.com/paper/2305.10758/) (2023)
- [Large Language Model Meets Graph Neural Network in Knowledge Distillation](https://mlsanity.com/paper/2402.05894/) (2024)
- [Weight-Inherited Distillation for Task-Agnostic BERT Compression](https://mlsanity.com/paper/2305.09098/) (2023)
- [AdaGMLP: AdaBoosting GNN-to-MLP Knowledge Distillation](https://mlsanity.com/paper/2405.14307/) (2024)
- [LoCa: Logit Calibration for Knowledge Distillation](https://mlsanity.com/paper/2409.04778/) (2024)
- [Propagate & Distill: Towards Effective Graph Learners Using Propagation-Embracing MLPs](https://mlsanity.com/paper/2311.17781/) (2023)
- [Unveiling the Unseen Potential of Graph Learning through MLPs: Effective Graph Learners Using Propagation-Embracing MLPs](https://mlsanity.com/paper/2311.11759/) (2023)

## Abstract (arXiv)

Distilling high-accuracy Graph Neural Networks (GNNs) to low-latency multilayer perceptions (MLPs) on graph tasks has become a hot research topic. However, conventional MLP learning relies almost exclusively on graph nodes and fails to effectively capture the graph structural information. Previous methods address this issue by processing graph edges into extra inputs for MLPs, but such graph structures may be unavailable for various scenarios. To this end, we propose Prototype-Guided Knowledge Distillation (PGKD), which does not require graph edges (edge-free setting) yet learns structure-aware MLPs. Our insight is to distill graph structural information from GNNs. Specifically, we first employ the class prototypes to analyze the impact of graph structures on GNN teachers, and then design two losses to distill such information from GNNs to MLPs. Experimental results on popular graph benchmarks demonstrate the effectiveness and robustness of the proposed PGKD.