mlsanityData to 7 Oct 2026

Diffusion-Assisted Distillation for Self-Supervised Graph Representation Learning with MLPs

arXiv 2510.04241 ↗First posted 5 Oct 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

For large-scale applications, there is growing interest in replacing Graph Neural Networks (GNNs) with lightweight Multi-Layer Perceptrons (MLPs) via knowledge distillation. However, distilling GNNs for self-supervised graph representation learning into MLPs is more challenging. This is because the performance of self-supervised learning is more related to the model's inductive bias than supervised learning. This motivates us to design a new distillation method to bridge a huge capacity gap between GNNs and MLPs in self-supervised graph representation learning. In this paper, we propose Diffusion-Assisted Distillation for Self-supervised Graph representation learning with MLPs (DAD-SGM). The proposed method employs a denoising diffusion model as a teacher assistant to better distill the knowledge from the teacher GNN into the student MLP. This approach enhances the generalizability and robustness of MLPs in self-supervised graph representation learning. Extensive experiments demonstrate that DAD-SGM effectively distills the knowledge of self-supervised GNNs compared to state-of-the-art GNN-to-MLP distillation methods. Our implementation is available at https://github.com/SeongJinAhn/DAD-SGM.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 5 Oct 2025.

Builds on

41 of 50 references.

References · 41

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2017Graph Attention Networkscited by 8,854
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  12. 2014DeepWalk: Online Learning of Social Representationscited by 1,543
29 more
  1. 2016Variational Graph Auto-Encoderscited by 1,373
  2. 2018Deep Graph Infomaxcited by 1,101
  3. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  4. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 987
  5. 2019Contrastive Representation Distillationcited by 734
  6. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 666
  7. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  8. 2020Deep Graph Contrastive Representation Learningcited by 535
  9. 2020Graph Contrastive Learning with Adaptive Augmentationcited by 519
  10. 2019Similarity-Preserving Knowledge Distillationcited by 507
  11. 2023Denoising Diffusion Models for Plug-and-Play Image Restorationcited by 335
  12. 2020Graph Random Neural Network for Semi-Supervised Learning on Graphscited by 213
  13. 2020Data Augmentation for Graph Neural Networkscited by 200
  14. 2022SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentationcited by 171
  15. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
  16. 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 138
  17. 2023Noise-Free Score Distillationcited by 99
  18. 2021Augmentation-Free Self-Supervised Learning on Graphscited by 94
  19. 2021Graph Barlow Twins: A self-supervised representation learning framework for graphscited by 85
  20. 2022Linkless Link Prediction via Relational Distillationcited by 39
  21. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  22. 2023Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Frameworkcited by 24
  23. 2022NOSMOG: Learning Noise-robust and Structure-aware MLPs on Graphscited by 14
  24. 2024AMD: Automatic Multi-step Distillation of Large-scale Vision Modelscited by 12
  25. 2020An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillationcited by 11
  26. 2022Unsupervised Network Embedding Beyond Homophilycited by 7
  27. 2024Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillationcited by 4
  28. 2023Diffusion Probabilistic Models for Structured Node Classificationcited by 3
  29. 2023Understanding the Effect of Data Augmentation on Knowledge Distillationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation
  2. 2025Simple Graph Contrastive Learning via Fractional-order Neural Diffusion Networks
  3. 2024Diffusing to the Top: Boost Graph Neural Networks with Minimal Hyperparameter Tuning
  4. 2022SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP
  5. 2022Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classification
  6. 2021Self-Distilled Self-Supervised Representation Learning
  7. 2025Diffusion As Self-Distillation: End-to-End Latent Diffusion In One Model
  8. 2025DiffGraph: Heterogeneous Graph Diffusion Model