mlsanityData to 7 Oct 2026

Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillation

arXiv 2407.14768 ↗First posted 20 Jul 2024cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

To bridge the gaps between powerful Graph Neural Networks (GNNs) and lightweight Multi-Layer Perceptron (MLPs), GNN-to-MLP Knowledge Distillation (KD) proposes to distill knowledge from a well-trained teacher GNN into a student MLP. In this paper, we revisit the knowledge samples (nodes) in teacher GNNs from the perspective of hardness, and identify that hard sample distillation may be a major performance bottleneck of existing graph KD algorithms. The GNN-to-MLP KD involves two different types of hardness, one student-free knowledge hardness describing the inherent complexity of GNN knowledge, and the other student-dependent distillation hardness describing the difficulty of teacher-to-student distillation. However, most of the existing work focuses on only one of these aspects or regards them as one thing. This paper proposes a simple yet effective Hardness-aware GNN-to-MLP Distillation (HGMD) framework, which decouples the two hardnesses and estimates them using a non-parametric approach. Finally, two hardness-aware distillation schemes (i.e., HGMD-weight and HGMD-mixup) are further proposed to distill hardness-aware knowledge from teacher GNNs into the corresponding nodes of student MLPs. As non-parametric distillation, HGMD does not involve any additional learnable parameters beyond the student MLPs, but it still outperforms most of the state-of-the-art competitors. HGMD-mixup improves over the vanilla MLPs by 12.95% and outperforms its teacher GNNs by 2.48% averaged over seven real-world datasets.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 27 of its 41 references that are in the map. First posted on arXiv 20 Jul 2024.

Builds on

27 of 41 references.

References · 27

  1. 2017Graph Attention Networkscited by 8,854
  2. 2017Inductive Representation Learning on Large Graphscited by 6,284
  3. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  4. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  5. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
  6. 2018Pitfalls of Graph Neural Network Evaluationcited by 994
  7. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  8. 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 544
  9. 2019MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixingcited by 442
  10. 2021Does Knowledge Distillation Really Work?cited by 187
  11. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
  12. 2021Self-supervised Learning on Graphs: Contrastive, Generative,or Predictivecited by 136
15 more
  1. 2020Distilling Knowledge from Graph Convolutional Networkscited by 128
  2. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  3. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 82
  4. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Frameworkcited by 69
  5. 2021Annealing Knowledge Distillationcited by 61
  6. 2021Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Studycited by 50
  7. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  8. 2020On Self-Distilling Graph Neural Networkcited by 35
  9. 2023Quantifying the Knowledge in GNNs for Reliable Distillation into MLPscited by 25
  10. 2023Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Frameworkcited by 24
  11. 2022FreeKD: Free-direction Knowledge Distillation for Graph Neural Networkscited by 22
  12. 2020Iterative Graph Self-Distillationcited by 21
  13. 2022Compressing Deep Graph Neural Networks via Adversarial Knowledge Distillationcited by 18
  14. 2021Multi-task Self-distillation for Graph-based Semi-Supervised Learningcited by 8
  15. 2024A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025ProGMLP: A Progressive Framework for GNN-to-MLP Knowledge Distillation with Efficient Trade-offscited by 1
  2. 2026InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillationcited by 1
  3. 2025Heuristic Methods are Good Teachers to Distill MLPs for Graph Link Predictioncited by 0
  4. 2025Diffusion-Assisted Distillation for Self-Supervised Graph Representation Learning with MLPscited by 0