mlsanityData to 7 Oct 2026

Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks

arXiv 2210.13014 ↗First posted 24 Oct 2022cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

We study a new paradigm of knowledge transfer that aims at encoding graph topological information into graph neural networks (GNNs) by distilling knowledge from a teacher GNN model trained on a complete graph to a student GNN model operating on a smaller or sparser graph. To this end, we revisit the connection between thermodynamics and the behavior of GNN, based on which we propose Neural Heat Kernel (NHK) to encapsulate the geometric property of the underlying manifold concerning the architecture of GNNs. A fundamental and principled solution is derived by aligning NHKs on teacher and student models, dubbed as Geometric Knowledge Distillation. We develop non- and parametric instantiations and demonstrate their efficacy in various experimental settings for knowledge distillation regarding different types of privileged topological information and teacher-student schemes.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 45 of its 74 references that are in the map. First posted on arXiv 24 Oct 2022.

Builds on

45 of 74 references.

References · 45

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2017Graph Attention Networkscited by 8,854
  4. 2017Inductive Representation Learning on Large Graphscited by 6,284
  5. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  7. 2020Knowledge Distillation: A Surveycited by 1,578
  8. 2019Simplifying Graph Convolutional Networkscited by 1,470
  9. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  10. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  11. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  12. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 987
33 more
  1. 2019Relational Knowledge Distillationcited by 873
  2. 2018Born Again Neural Networkscited by 539
  3. 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 471
  4. 2016Geometric deep learning on graphs and manifolds using mixture model CNNscited by 468
  5. 2013Do Deep Nets Really Need to be Deep?cited by 466
  6. 2015Deep Kernel Learningcited by 440
  7. 2019Diffusion Improves Graph Learningcited by 414
  8. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  9. 2021GRAND: Graph Neural Diffusioncited by 258
  10. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  11. 2015Unifying distillation and privileged informationcited by 218
  12. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 218
  13. 2020Generalization and Representational Limits of Graph Neural Networkscited by 217
  14. 2022Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNscited by 208
  15. 2020Graph-Bert: Only Attention is Needed for Learning Graph Representationscited by 176
  16. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  17. 2022Handling Distribution Shifts on Graphs: An Invariance Perspectivecited by 163
  18. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
  19. 2021Subgraph Federated Learning with Missing Neighbor Generationcited by 132
  20. 2020Distilling Knowledge from Graph Convolutional Networkscited by 128
  21. 2021PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equationscited by 106
  22. 2021Beltrami Flow and Neural Diffusion on Graphscited by 84
  23. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Frameworkcited by 69
  24. 2021Dissecting the Diffusion Process in Linear Graph Convolutional Networkscited by 58
  25. 2018Graph Coarsening with Preserved Spectral Propertiescited by 41
  26. 2020On Self-Distilling Graph Neural Networkcited by 35
  27. 2014Heat kernel based community detectioncited by 30
  28. 2022Cross-Task Knowledge Distillation in Multi-Task Recommendationcited by 22
  29. 2021Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approachcited by 20
  30. 2020Faster Graph Embeddings via Coarseningcited by 15
  31. 2019Personalized Graph Neural Networks with Attention Mechanism for Session-Aware Recommendationcited by 10
  32. 2021Privileged Graph Distillation for Cold Start Recommendationcited by 10
  33. 2013The nonlinear heat equation on dense graphs and graph limitscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 9; 2025: 2; 2026: 3.

22 papers.

Sort:

By area:

  1. 2023NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classificationcited by 207
  2. 2023SGFormer: Simplifying and Empowering Transformers for Large-Graph Representationscited by 131
  3. 2023DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusioncited by 85
  4. 2023Energy-based Out-of-Distribution Detection for Graph Neural Networkscited by 67
  5. 2024LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddingscited by 60
  6. 2022Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPscited by 42
  7. 2023Knowledge Distillation on Graphs: A Surveycited by 38
  8. 2025Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometriescited by 15
  9. 2023GraphGLOW: Universal and Generalizable Structure Learning for Graph Neural Networkscited by 12
  10. 2024Node Identifiers: Compact, Discrete Representations for Efficient Graph Learningcited by 10
  11. 2023Accelerating Scalable Graph Neural Network Inference with Node-Adaptive Propagationcited by 8
  12. 2024Mitigating Label Noise on Graph via Topological Sample Selectioncited by 6
10 more
  1. 2024EasyST: A Simple Framework for Spatio-Temporal Predictioncited by 6
  2. 2025GraphSOS: Graph Sampling and Order Selection to Help LLMs Understand Graphs Bettercited by 4
  3. 2024Learning Divergence Fields for Shift-Robust Graph Representationscited by 2
  4. 2026An LLM-Guided Query-Aware Inference System for GNN Models on Large Knowledge Graphscited by 2
  5. 2024Introducing Diminutive Causal Structure into Graph Representation Learningcited by 1
  6. 2024Transformers from Diffusion: A Unified Framework for Neural Message Passingcited by 1
  7. 2024Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Frameworkcited by 0
  8. 2024Acceleration Algorithms in GNNs: A Surveycited by 0
  9. 2026LoReC: Rethinking Large Language Models for Graph Data Analysiscited by 0
  10. 2026Model-to-Data Distillation for Graph Neural Networkscited by 0