mlsanityData to 7 Oct 2026

Distilling Self-Knowledge From Contrastive Links to Classify Graph Nodes Without Passing Messages

arXiv 2106.08541 ↗First posted 16 Jun 2021cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Nowadays, Graph Neural Networks (GNNs) following the Message Passing paradigm become the dominant way to learn on graphic data. Models in this paradigm have to spend extra space to look up adjacent nodes with adjacency matrices and extra time to aggregate multiple messages from adjacent nodes. To address this issue, we develop a method called LinkDist that distils self-knowledge from connected node pairs into a Multi-Layer Perceptron (MLP) without the need to aggregate messages. Experiment with 8 real-world datasets shows the MLP derived from LinkDist can predict the label of a node without knowing its adjacencies but achieve comparable accuracy against GNNs in the contexts of semi- and full-supervised node classification. Moreover, LinkDist benefits from its Non-Message Passing paradigm that we can also distil self-knowledge from arbitrarily sampled node pairs in a contrastive way to further boost the performance of LinkDist.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 15 of its 28 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

15 of 28 references.

References · 15

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2017Inductive Representation Learning on Large Graphscited by 6,284
  5. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  6. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  7. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
  8. 2020Knowledge Distillation: A Surveycited by 1,578
  9. 2015Image-based Recommendations on Styles and Substitutescited by 702
  10. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  11. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 583
  12. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
3 more
  1. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  2. 2017Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Rankingcited by 332
  3. 2020Regularizing Class-wise Predictions via Self-knowledge Distillationcited by 144

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 3; 2024: 4.

11 papers.

  1. 2021Local Augmentation for Graph Neural Networkscited by 52
  2. 2023Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Frameworkcited by 24
  3. 2024A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillationcited by 8
  4. 2024Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Taskscited by 7
  5. 2022Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classificationcited by 6
  6. 2022Automated Graph Self-supervised Learning via Multi-teacher Knowledge Distillationcited by 6
  7. 2023Graph-based Knowledge Distillation: A survey and experimental evaluationcited by 6
  8. 2023Clarify Confused Nodes via Separated Learningcited by 5
  9. 2024Graph Neural Networks Need Cluster-Normalize-Activate Modulescited by 2
  10. 2022Graph Decipher: A transparent dual-attention graph neural network to understand the message-passing mechanism for the node classificationcited by 0
  11. 2024Learning to Model Graph Structural Information on MLPs via Graph Structure Self-Contrastingcited by 0