mlsanityData to 14 Oct 2026

Frameless Graph Knowledge Distillation

arXiv 2307.06631 ↗First posted 13 Jul 2023cs.LGcited by 2

On the map: Knowledge distillation → Graph neural network distillation

Abstract (arXiv)

Knowledge distillation (KD) has shown great potential for transferring knowledge from a complex teacher model to a simple student model in which the heavy learning task can be accomplished efficiently and without losing too much prediction accuracy. Recently, many attempts have been made by applying the KD mechanism to the graph representation learning models such as graph neural networks (GNNs) to accelerate the model's inference speed via student models. However, many existing KD-based GNNs utilize MLP as a universal approximator in the student model to imitate the teacher model's process without considering the graph knowledge from the teacher model. In this work, we provide a KD-based framework on multi-scaled GNNs, known as graph framelet, and prove that by adequately utilizing the graph knowledge in a multi-scaled manner provided by graph framelet decomposition, the student model is capable of adapting both homophilic and heterophilic graphs and has the potential of alleviating the over-squashing issue with a simple yet effectively graph surgery. Furthermore, we show how the graph knowledge supplied by the teacher is learned and digested by the student model via both algebra and geometry. Comprehensive experiments show that our proposed model can generate learning accuracy identical to or even surpass the teacher model while maintaining the high speed of inference.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation. Builds on 59 of its 94 references that are in the map. First posted on arXiv 13 Jul 2023.

Builds on

59 of 94 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP18 Oct

Other references · 58

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2017Graph Attention Networkscited by 8,923
  5. 2017Inductive Representation Learning on Large Graphscited by 6,320
  6. 2018How Powerful are Graph Neural Networks?cited by 4,393
  7. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  8. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  9. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,089
  10. 2020Knowledge Distillation: A Surveycited by 1,603
  11. 2019Simplifying Graph Convolutional Networkscited by 1,477
  12. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
38 more
  1. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  2. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,102
  3. 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 826
  4. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 734
  5. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 680
  6. 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643
  7. 2021Score-based Generative Modeling in Latent Spacecited by 636
  8. 2017Deep Mutual Learningcited by 628
  9. 2020Adaptive Universal Generalized PageRank Graph Neural Networkcited by 595
  10. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  11. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  12. 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 478
  13. 2016FastText.zip: Compressing text classification modelscited by 335
  14. 2019Variational Information Distillation for Knowledge Transfercited by 324
  15. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 257
  16. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  17. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  18. 2019Learning Lightweight Lane Detection CNNs by Self Attention Distillationcited by 162
  19. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 152
  20. 2019Graph Wavelet Neural Networkcited by 131
  21. 2020Distilling Knowledge from Graph Convolutional Networkscited by 129
  22. 2019Online Knowledge Distillation with Diverse Peerscited by 128
  23. 2020Degree-Quant: Quantization-Aware Training for Graph Neural Networkscited by 96
  24. 2020Explaining Knowledge Distillation by Quantifying the Knowledgecited by 66
  25. 2022Understanding convolution on graphs via energiescited by 59
  26. 2018Light Gated Recurrent Units for Speech Recognitioncited by 48
  27. 2021How Framelets Enhance Graph Neural Networkscited by 46
  28. 2021Accelerating Large Scale Real-Time GNN Inference using Channel Pruningcited by 41
  29. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  30. 2020On Self-Distilling Graph Neural Networkcited by 35
  31. 2020Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teachercited by 33
  32. 2019Deep geometric knowledge distillation with graphscited by 21
  33. 2019Fast Haar Transforms for Graph Neural Networkscited by 20
  34. 2020Manifold structure in graph embeddingscited by 17
  35. 2020Decimated Framelet System on Graphs and Fast G-Framelet Transformscited by 17
  36. 2019Self-supervised Label Augmentation via Input Transformationscited by 15
  37. 2018Network Representation Using Graph Root Distributionscited by 12
  38. 2022Skeleton-based Action Recognition via Adaptive Cross-Form Learningcited by 12

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Design Your Own Universe: A Physics-Informed Agnostic Method for Enhancing Graph Neural Networkscited by 6
  2. 2024When Graph Neural Networks Meet Dynamic Mode Decompositioncited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2023Unveiling the Unseen Potential of Graph Learning through MLPs: Effective Graph Learners Using Propagation-Embracing MLPs
  2. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework
  3. 2024A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation
  4. 2023Knowledge Distillation on Graphs: A Survey
  5. 2022Boosting Graph Neural Networks via Adaptive Knowledge Distillation
  6. 2021Graph-Free Knowledge Distillation for Graph Neural Networks
  7. 2025Enhancing Graph Neural Networks: A Mutual Learning Approach
  8. 2023Propagate & Distill: Towards Effective Graph Learners Using Propagation-Embracing MLPs