mlsanityData to 7 Oct 2026

Knowledge Distillation on Graphs: A Survey

arXiv 2302.00219 ↗First posted 1 Feb 2023cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Graph Neural Networks (GNNs) have attracted tremendous attention by demonstrating their capability to handle graph data. However, they are difficult to be deployed in resource-limited devices due to model sizes and scalability constraints imposed by the multi-hop data dependency. In addition, real-world graphs usually possess complex structural information and features. Therefore, to improve the applicability of GNNs and fully encode the complicated topological information, knowledge distillation on graphs (KDG) has been introduced to build a smaller yet effective model and exploit more knowledge from data, leading to model compression and performance improvement. Recently, KDG has achieved considerable progress with many studies proposed. In this survey, we systematically review these works. Specifically, we first introduce KDG challenges and bases, then categorize and summarize existing works of KDG by answering the following three questions: 1) what to distillate, 2) who to whom, and 3) how to distillate. Finally, we share our thoughts on future research directions.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 65 of its 107 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

65 of 107 references.

References · 65

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2017Graph Attention Networkscited by 8,854
  4. 2017Inductive Representation Learning on Large Graphscited by 6,284
  5. 2018How Powerful are Graph Neural Networks?cited by 4,355
  6. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  7. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  8. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  9. 2020Knowledge Distillation: A Surveycited by 1,578
  10. 2019When Does Label Smoothing Help?cited by 985
  11. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  12. 2019Relational Knowledge Distillationcited by 873
38 more
  1. 2018Hierarchical Graph Representation Learning with Differentiable Poolingcited by 786
  2. 2019Graph Neural Networks for Social Recommendationcited by 711
  3. 2020Knowledge Graphscited by 619
  4. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  5. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  6. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  7. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  8. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  9. 2020Graph Structure Learning for Robust Graph Neural Networkscited by 368
  10. 2019Variational Information Distillation for Knowledge Transfercited by 321
  11. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  12. 2021Graph Neural Network-Based Anomaly Detection in Multivariate Time Seriescited by 300
  13. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  14. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  15. 2021Graph Neural Network for Traffic Forecasting: A Surveycited by 260
  16. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  17. 2021Towards Understanding Knowledge Distillationcited by 158
  18. 2020Rumor Detection on Social Media with Bi-Directional Graph Convolutional Networkscited by 152
  19. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
  20. 2021Graph Condensation for Graph Neural Networkscited by 150
  21. 20213D Infomax improves GNNs for Molecular Property Predictioncited by 146
  22. 2020Distilling Knowledge from Graph Convolutional Networkscited by 128
  23. 2021A Unified Lottery Ticket Hypothesis for Graph Neural Networkscited by 118
  24. 2020Graphon Neural Networks and the Transferability of Graph Neural Networkscited by 117
  25. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Frameworkcited by 69
  26. 2021Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoodscited by 57
  27. 2022Graph-based Molecular Representation Learningcited by 51
  28. 2022Heterogeneous Graph Masked Autoencoderscited by 47
  29. 2021Graph-Free Knowledge Distillation for Graph Neural Networkscited by 46
  30. 2021Efficient Medical Image Segmentation Based on Knowledge Distillationcited by 43
  31. 2022Linkless Link Prediction via Relational Distillationcited by 39
  32. 2021On Representation Knowledge Distillation for Graph Neural Networkscited by 38
  33. 2020On Self-Distilling Graph Neural Networkcited by 35
  34. 2022T2-GNN: Graph Neural Networks for Graphs with Incomplete Features and Structure via Teacher-Student Distillationcited by 32
  35. 2022LTE4G: Long-Tail Experts for Graph Neural Networkscited by 30
  36. 2021Distilling Holistic Knowledge with Graph Neural Networkscited by 26
  37. 2022FreeKD: Free-direction Knowledge Distillation for Graph Neural Networkscited by 22
  38. 2022Geometric Knowledge Distillation: Topology Compression for Graph Neural Networkscited by 22

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 13; 2025: 8; 2026: 10.

38 papers.

Sort:

By area:

  1. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  2. 2023VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPscited by 34
  3. 2024The Heterophilic Graph Learning Handbook: Benchmarks, Models, Theoretical Analysis, Applications and Challengescited by 29
  4. 2023Data-centric Graph Learning: A Surveycited by 22
  5. 2023Graph Meets LLMs: Towards Large Graph Modelscited by 19
  6. 2024LLM and GNN are Complementary: Distilling LLM for Multimodal Graph Learningcited by 17
  7. 2023Accelerating Molecular Graph Neural Networks via Knowledge Distillationcited by 11
  8. 2024A Survey on Multimodal Wearable Sensor-based Human Action Recognitioncited by 10
  9. 2024Node Identifiers: Compact, Discrete Representations for Efficient Graph Learningcited by 10
  10. 2025Automated Knowledge Graph Construction using Large Language Models and Sentence Complexity Modellingcited by 9
  11. 2024A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillationcited by 8
  12. 2024BadCLM: Backdoor Attack in Clinical Language Models for Electronic Health Recordscited by 5
26 more
  1. 2023Towards the Fundamental Limits of Knowledge Transfer over Finite Domainscited by 4
  2. 2024TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillationcited by 4
  3. 2024Classifying Nodes in Graphs without GNNscited by 2
  4. 2024UGMAE: A Unified Framework for Graph Masked Autoencoderscited by 2
  5. 2025Intellectual Property in Graph-Based Machine Learning as a Service: Attacks and Defensescited by 2
  6. 2026DSBD: Dual-Aligned Structural Basis Distillation for Graph Domain Adaptationcited by 2
  7. 2023STRIDE: Structure and Embedding Distillation with Attention for Graph Neural Networkscited by 1
  8. 2025Towards Faster and More Compact Foundation Models for Molecular Property Predictioncited by 1
  9. 2025HERO: Preserving Structure and Semantics in Heterogeneous Continual Graph Learningcited by 1
  10. 2025Learning from Random Subspace Exploration: Generalized Test-Time Augmentation with Self-supervised Distillationcited by 1
  11. 2026PRECTR-V2:Unified Relevance-CTR Framework with Cross-User Preference Mining, Exposure Bias Correction, and LLM-Distilled Encoder Optimizationcited by 1
  12. 2026Pruned Traffic Trees: Native Semantic Compression with a Protocol-Structured Model Family for Encrypted Traffic Classificationcited by 1
  13. 2023LIB-KD: Teaching Inductive Bias for Efficient Vision Transformer Distillation and Compressioncited by 0
  14. 2024FlyKD: Graph Knowledge Distillation on the Fly with Curriculum Learningcited by 0
  15. 2024Acceleration Algorithms in GNNs: A Surveycited by 0
  16. 2024Preserving Node Distinctness in Graph Autoencoders via Similarity Distillationcited by 0
  17. 2025Reconciling Attribute and Structural Anomalies for Improved Graph Anomaly Detectioncited by 0
  18. 2025Efficient Graph Knowledge Distillation from GNNs to Kolmogorov--Arnold Networks via Self-Attention Dynamic Samplingcited by 0
  19. 2025Layer-to-Layer Knowledge Mixing in Graph Neural Network for Chemical Property Predictioncited by 0
  20. 2026S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Modelscited by 0
  21. 2026The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Workscited by 0
  22. 2026Generalist Graph Anomaly Detection via Prototype-Based Distillationcited by 0
  23. 2026Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillationcited by 0
  24. 2026Backdoor Learning in Language Models and Vision-Language Modelscited by 0
  25. 2026Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillationcited by 0
  26. 2026Data-Efficient Agentic Graph Domain Adaptation via Reliability-Aware Prototype Learningcited by 0