# Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks

arXiv: https://arxiv.org/abs/2210.13014 · HTML: https://mlsanity.com/paper/2210.13014/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- Builds on 45 of its 74 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Builds on (most cited first, 30 of 45)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Predict then Propagate: Graph Neural Networks meet Personalized PageRank](https://mlsanity.com/paper/1810.05997/) (2018)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Understanding over-squashing and bottlenecks on graphs via curvature](https://mlsanity.com/paper/2111.14522/) (2021)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Geometric deep learning on graphs and manifolds using mixture model CNNs](https://mlsanity.com/paper/1611.08402/) (2016)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Diffusion Improves Graph Learning](https://mlsanity.com/paper/1911.05485/) (2019)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [GRAND: Graph Neural Diffusion](https://mlsanity.com/paper/2106.10934/) (2021)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks](https://mlsanity.com/paper/2009.11848/) (2020)
- [Generalization and Representational Limits of Graph Neural Networks](https://mlsanity.com/paper/2002.06157/) (2020)
- [Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs](https://mlsanity.com/paper/2202.04579/) (2022)
- [Graph-Bert: Only Attention is Needed for Learning Graph Representations](https://mlsanity.com/paper/2001.05140/) (2020)
- [Self-Distillation Amplifies Regularization in Hilbert Space](https://mlsanity.com/paper/2002.05715/) (2020)
- [Handling Distribution Shifts on Graphs: An Invariance Perspective](https://mlsanity.com/paper/2202.02466/) (2022)
- [Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation](https://mlsanity.com/paper/2110.08727/) (2021)

## Built on by (most cited first, 22 of 22)

- [NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification](https://mlsanity.com/paper/2306.08385/) (2023)
- [SGFormer: Simplifying and Empowering Transformers for Large-Graph Representations](https://mlsanity.com/paper/2306.10759/) (2023)
- [DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion](https://mlsanity.com/paper/2301.09474/) (2023)
- [Energy-based Out-of-Distribution Detection for Graph Neural Networks](https://mlsanity.com/paper/2302.02914/) (2023)
- [LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings](https://mlsanity.com/paper/2408.14512/) (2024)
- [Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPs](https://mlsanity.com/paper/2212.09034/) (2022)
- [Knowledge Distillation on Graphs: A Survey](https://mlsanity.com/paper/2302.00219/) (2023)
- [Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries](https://mlsanity.com/paper/2504.08896/) (2025)
- [GraphGLOW: Universal and Generalizable Structure Learning for Graph Neural Networks](https://mlsanity.com/paper/2306.11264/) (2023)
- [Node Identifiers: Compact, Discrete Representations for Efficient Graph Learning](https://mlsanity.com/paper/2405.16435/) (2024)
- [Accelerating Scalable Graph Neural Network Inference with Node-Adaptive Propagation](https://mlsanity.com/paper/2310.10998/) (2023)
- [Mitigating Label Noise on Graph via Topological Sample Selection](https://mlsanity.com/paper/2403.01942/) (2024)
- [EasyST: A Simple Framework for Spatio-Temporal Prediction](https://mlsanity.com/paper/2409.06748/) (2024)
- [GraphSOS: Graph Sampling and Order Selection to Help LLMs Understand Graphs Better](https://mlsanity.com/paper/2501.14427/) (2025)
- [Learning Divergence Fields for Shift-Robust Graph Representations](https://mlsanity.com/paper/2406.04963/) (2024)
- [An LLM-Guided Query-Aware Inference System for GNN Models on Large Knowledge Graphs](https://mlsanity.com/paper/2603.04545/) (2026)
- [Introducing Diminutive Causal Structure into Graph Representation Learning](https://mlsanity.com/paper/2406.08709/) (2024)
- [Transformers from Diffusion: A Unified Framework for Neural Message Passing](https://mlsanity.com/paper/2409.09111/) (2024)
- [Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework](https://mlsanity.com/paper/2403.01079/) (2024)
- [Acceleration Algorithms in GNNs: A Survey](https://mlsanity.com/paper/2405.04114/) (2024)
- [LoReC: Rethinking Large Language Models for Graph Data Analysis](https://mlsanity.com/paper/2604.17897/) (2026)
- [Model-to-Data Distillation for Graph Neural Networks](https://mlsanity.com/paper/2605.06814/) (2026)

## Abstract (arXiv)

We study a new paradigm of knowledge transfer that aims at encoding graph topological information into graph neural networks (GNNs) by distilling knowledge from a teacher GNN model trained on a complete graph to a student GNN model operating on a smaller or sparser graph. To this end, we revisit the connection between thermodynamics and the behavior of GNN, based on which we propose Neural Heat Kernel (NHK) to encapsulate the geometric property of the underlying manifold concerning the architecture of GNNs. A fundamental and principled solution is derived by aligning NHKs on teacher and student models, dubbed as Geometric Knowledge Distillation. We develop non- and parametric instantiations and demonstrate their efficacy in various experimental settings for knowledge distillation regarding different types of privileged topological information and teacher-student schemes.