mlsanityData to 7 Oct 2026

Graph-based Knowledge Distillation: A survey and experimental evaluation

arXiv 2302.14643 ↗First posted 27 Feb 2023cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Graph, such as citation networks, social networks, and transportation networks, are prevalent in the real world. Graph Neural Networks (GNNs) have gained widespread attention for their robust expressiveness and exceptional performance in various graph applications. However, the efficacy of GNNs is heavily reliant on sufficient data labels and complex network models, with the former obtaining hardly and the latter computing costly. To address the labeled data scarcity and high complexity of GNNs, Knowledge Distillation (KD) has been introduced to enhance existing GNNs. This technique involves transferring the soft-label supervision of the large teacher model to the small student model while maintaining prediction performance. This survey offers a comprehensive overview of Graph-based Knowledge Distillation methods, systematically categorizing and summarizing them while discussing their limitations and future directions. This paper first introduces the background of graph and KD. It then provides a comprehensive summary of three types of Graph-based Knowledge Distillation methods, namely Graph-based Knowledge Distillation for deep neural networks (DKD), Graph-based Knowledge Distillation for GNNs (GKD), and Self-Knowledge Distillation based Graph-based Knowledge Distillation (SKD). Each type is further divided into knowledge distillation methods based on the output layer, middle layer, and constructed graph. Subsequently, various algorithms' ideas are analyzed and compared, concluding with the advantages and disadvantages of each algorithm supported by experimental results. In addition, the applications of graph-based knowledge distillation in CV, NLP, RS, and other fields are listed. Finally, the graph-based knowledge distillation is summarized and prospectively discussed. We have also released related resources at https://github.com/liujing1023/Graph-based-Knowledge-Distillation.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 93 of its 134 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

93 of 134 references.

References · 93

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2017Graph Attention Networkscited by 8,854
  6. 2017Inductive Representation Learning on Large Graphscited by 6,284
  7. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  8. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  9. 2017Non-local Neural Networkscited by 2,290
  10. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,085
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  12. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
38 more
  1. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  2. 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,097
  3. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  4. 2018Pitfalls of Graph Neural Network Evaluationcited by 994
  5. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 970
  6. 2019Relational Knowledge Distillationcited by 873
  7. 2018Link Prediction Based on Graph Neural Networkscited by 829
  8. 2019Graph Neural Networks for Social Recommendationcited by 711
  9. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  10. 2017Deep Mutual Learningcited by 621
  11. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 583
  12. 2018Born Again Neural Networkscited by 539
  13. 2019Similarity-Preserving Knowledge Distillationcited by 507
  14. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  15. 2020Graph Neural Networks in Recommender Systems: A Surveycited by 493
  16. 2018Deep Learning on Graphs: A Surveycited by 393
  17. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  18. 2020Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloudcited by 335
  19. 2019Correlation Congruence for Knowledge Distillationcited by 280
  20. 2015Diffusion-Convolutional Neural Networkscited by 268
  21. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  22. 2019A Fair Comparison of Graph Neural Networks for Graph Classificationcited by 244
  23. 2019Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networkscited by 241
  24. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  25. 2019Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networkscited by 177
  26. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  27. 2018Adversarial Attack and Defense on Graph Data: A Surveycited by 166
  28. 2018Adaptive Graph Convolutional Neural Networkscited by 155
  29. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
  30. 2018Traffic Graph Convolutional Recurrent Neural Network: A Deep Learning Framework for Network-Scale Traffic Learning and Forecastingcited by 145
  31. 2019Graph Wavelet Neural Networkcited by 131
  32. 2020Distilling Knowledge from Graph Convolutional Networkscited by 128
  33. 2020Foundations and modelling of dynamic networks using Dynamic Graph Neural Networks: A surveycited by 123
  34. 2020Spatio-Temporal Graph for Video Captioning with Knowledge Distillationcited by 109
  35. 2020Graph Neural Networks Meet Neural-Symbolic Computing: A Survey and Perspectivecited by 93
  36. 2020Heterogeneous Knowledge Distillation using Information Flow Modelingcited by 73
  37. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Frameworkcited by 69
  38. 2021Deep Graph-level Anomaly Detection by Glocal Knowledge Distillationcited by 67

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3; 2025: 1.

6 papers.

  1. 2023Data-centric Graph Learning: A Surveycited by 22
  2. 2023Accelerating Molecular Graph Neural Networks via Knowledge Distillationcited by 11
  3. 2024Simple Graph Condensationcited by 8
  4. 2024Edge Graph Intelligence: Reciprocally Empowering Edge Networks with Graph Intelligencecited by 2
  5. 2024Training Self-localization Models for Unseen Unfamiliar Places via Teacher-to-Student Data-Free Knowledge Transfercited by 1
  6. 2025REACT-KD: Region-Aware Cross-modal Topological Knowledge Distillation for Interpretable Medical Image Classificationcited by 0