mlsanityData to 14 Oct 2026

Propagate & Distill: Towards Effective Graph Learners Using Propagation-Embracing MLPs

arXiv 2311.17781 ↗First posted 29 Nov 2023cs.LG cs.AI cs.ITcited by 1

On the map: Knowledge distillation → Graph neural network distillation

Abstract (arXiv)

Recent studies attempted to utilize multilayer perceptrons (MLPs) to solve semisupervised node classification on graphs, by training a student MLP by knowledge distillation from a teacher graph neural network (GNN). While previous studies have focused mostly on training the student MLP by matching the output probability distributions between the teacher and student models during distillation, it has not been systematically studied how to inject the structural information in an explicit and interpretable manner. Inspired by GNNs that separate feature transformation T and propagation Pi, we re-frame the distillation process as making the student MLP learn both T and Pi. Although this can be achieved by applying the inverse propagation Pi-1 before distillation from the teacher, it still comes with a high computational cost from large matrix multiplications during training. To solve this problem, we propose Propagate & Distill (P&D), which propagates the output of the teacher before distillation, which can be interpreted as an approximate process of the inverse propagation. We demonstrate that P&D can readily improve the performance of the student MLP.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation. Builds on 26 of its 37 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

26 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs24 Mar
  2. 2022SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP18 Oct
  3. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework4 Mar

Other references · 23

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2017Inductive Representation Learning on Large Graphscited by 6,320
  3. 2018How Powerful are Graph Neural Networks?cited by 4,393
  4. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  5. 2020Knowledge Distillation: A Surveycited by 1,603
  6. 2014DeepWalk: Online Learning of Social Representationscited by 1,547
  7. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,102
  8. 2018Pitfalls of Graph Neural Network Evaluationcited by 1,007
  9. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 993
  10. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 976
  11. 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 881
  12. 2020Adaptive Universal Generalized PageRank Graph Neural Networkcited by 595
11 more
  1. 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 545
  2. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  3. 2020Combining Label Propagation and Simple Models Out-performs Graph Neural Networkscited by 202
  4. 2020Scaling Graph Neural Networks with Approximate PageRankcited by 174
  5. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 152
  6. 2020A Unified View on Graph Neural Networks as Graph Signal Denoisingcited by 110
  7. 2020Implicit Graph Neural Networkscited by 100
  8. 2021Graph-MLP: Node Classification without Message Passing in Graphcited by 73
  9. 2021Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoodscited by 58
  10. 2023Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Frameworkcited by 24
  11. 2022Node Representation Learning in Graph via Node-to-Neighbourhood Mutual Information Maximizationcited by 9

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Acceleration Algorithms in GNNs: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Unveiling the Unseen Potential of Graph Learning through MLPs: Effective Graph Learners Using Propagation-Embracing MLPs
  2. 2026InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillation
  3. 2024A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation
  4. 2025Heuristic Methods are Good Teachers to Distill MLPs for Graph Link Prediction
  5. 2023Frameless Graph Knowledge Distillation
  6. 2023Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs
  7. 2023VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs
  8. 2024Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework