mlsanityData to 14 Oct 2026

Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework

arXiv 2403.01079 ↗First posted 2 Mar 2024cs.LG cs.AIcited by 0

On the map: Knowledge distillation → Graph neural network distillation

Abstract (arXiv)

We study the challenging problem for inference tasks on large-scale graph datasets of Graph Neural Networks: huge time and memory consumption, and try to overcome it by reducing reliance on graph structure. Even though distilling graph knowledge to student MLP is an excellent idea, it faces two major problems of positional information loss and low generalization. To solve the problems, we propose a new three-stage multitask distillation framework. In detail, we use Positional Encoding to capture positional information. Also, we introduce Neural Heat Kernels responsible for graph data processing in GNN and utilize hidden layer outputs matching for better performance of student MLP's hidden layers. To the best of our knowledge, it is the first work to include hidden layer distillation for student MLP on graphs and to combine graph Positional Encoding with MLP. We test its performance and robustness with several settings and draw the conclusion that our work can outperform well with good stability. Code implementation is in https://github.com/lijunxian111/KMP.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation. Builds on 37 of its 45 references that are in the map. First posted on arXiv 2 Mar 2024.

Builds on

37 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks24 Oct
  2. 2022SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP18 Oct
  3. 2022Compressing Deep Graph Neural Networks via Adversarial Knowledge Distillation24 May
  4. 2021On Representation Knowledge Distillation for Graph Neural Networks9 Nov
  5. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework4 Mar

Other references · 32

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2017Graph Attention Networkscited by 8,923
  4. 2018How Powerful are Graph Neural Networks?cited by 4,393
  5. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  6. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  8. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  9. 2019Simplifying Graph Convolutional Networkscited by 1,477
  10. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,139
  11. 2019Graph Neural Networks for Social Recommendationcited by 714
  12. 2020A Generalization of Transformer Networks to Graphscited by 661
20 more
  1. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 541
  2. 2019Graph Transformer Networkscited by 523
  3. 2020Graph Neural Networks in Recommender Systems: A Surveycited by 494
  4. 2013Do Deep Nets Really Need to be Deep?cited by 474
  5. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 250
  6. 2016Molecular Graph Convolutions: Moving Beyond Fingerprintscited by 238
  7. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  8. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 152
  9. 2020Distilling Knowledge from Graph Convolutional Networkscited by 129
  10. 2020Degree-Quant: Quantization-Aware Training for Graph Neural Networkscited by 96
  11. 2022Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networkscited by 94
  12. 2021Graph-MLP: Node Classification without Message Passing in Graphcited by 73
  13. 2020Graph Convolutional Networks using Heat Kernel for Semi-supervised Learningcited by 51
  14. 2021Accelerating Large Scale Real-Time GNN Inference using Channel Pruningcited by 41
  15. 2020On Self-Distilling Graph Neural Networkcited by 35
  16. 2019Graph-based Knowledge Distillation by Multi-head Attention Networkcited by 31
  17. 2022Survey on Graph Neural Network Acceleration: An Algorithmic Perspectivecited by 26
  18. 2016Learning Sparse Graphs Under Smoothness Priorcited by 23
  19. 2019Deep geometric knowledge distillation with graphscited by 21
  20. 2021Reinforcement Learning For Data Poisoning on Graph Neural Networkscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation
  2. 2023Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework
  3. 2023Unveiling the Unseen Potential of Graph Learning through MLPs: Effective Graph Learners Using Propagation-Embracing MLPs
  4. 2026Distilling Graph Geometry: Knowledge Gap from GNNs to MLPs
  5. 2026InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillation
  6. 2023Quantifying the Knowledge in GNNs for Reliable Distillation into MLPs
  7. 2023Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs
  8. 2024Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillation