SimMLP: Training MLPs on Graphs without Supervision
Not on the map. Why?
Abstract (arXiv)
Graph Neural Networks (GNNs) have demonstrated their effectiveness in various graph learning tasks, yet their reliance on neighborhood aggregation during inference poses challenges for deployment in latency-sensitive applications, such as real-time financial fraud detection. To address this limitation, recent studies have proposed distilling knowledge from teacher GNNs into student Multi-Layer Perceptrons (MLPs) trained on node content, aiming to accelerate inference. However, these approaches often inadequately explore structural information when inferring unseen nodes. To this end, we introduce SimMLP, a Self-supervised framework for learning MLPs on graphs, designed to fully integrate rich structural information into MLPs. Notably, SimMLP is the first MLP-learning method that can achieve equivalence to GNNs in the optimal case. The key idea is to employ self-supervised learning to align the representations encoded by graph context-aware GNNs and neighborhood dependency-free MLPs, thereby fully integrating the structural information into MLPs. We provide a comprehensive theoretical analysis, demonstrating the equivalence between SimMLP and GNNs based on mutual information and inductive bias, highlighting SimMLP's advanced structural learning capabilities. Additionally, we conduct extensive experiments on 20 benchmark datasets, covering node classification, link prediction, and graph classification, to showcase SimMLP's superiority over state-of-the-art baselines, particularly in scenarios involving unseen nodes (e.g., inductive and cold-start node classification) where structural insights are crucial. Our codes are available at: https://github.com/Zehong-Wang/SimMLP.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 54 of its 89 references that are in the map. First posted on arXiv 14 Feb 2024.
Builds on
54 of 89 references.
References · 54
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2017Graph Attention Networkscited by 8,854
- 2017Inductive Representation Learning on Large Graphscited by 6,284
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2017Neural Discrete Representation Learningcited by 4,650
- 2018How Powerful are Graph Neural Networks?cited by 4,355
- 2017Neural Message Passing for Quantum Chemistrycited by 3,460
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
38 more
- 2019Simplifying Graph Convolutional Networkscited by 1,470
- 2020Graph Contrastive Learning with Augmentationscited by 1,203
- 2018Deep Graph Infomaxcited by 1,101
- 2018Pitfalls of Graph Neural Network Evaluationcited by 994
- 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 970
- 2016Deep Variational Information Bottleneckcited by 915
- 2020Simple and Deep Graph Convolutional Networkscited by 857
- 2020What Makes for Good Views for Contrastive Learning?cited by 843
- 2015Deep Learning and the Information Bottleneck Principlecited by 793
- 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
- 2020TUDataset: A collection of benchmark datasets for learning with graphscited by 718
- 2020Contrastive Multi-View Representation Learning on Graphscited by 651
- 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 544
- 2020Deep Graph Contrastive Representation Learningcited by 535
- 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
- 2020Graph Contrastive Learning with Adaptive Augmentationcited by 519
- 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
- 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
- 2022GraphMAE: Self-Supervised Masked Graph Autoencoderscited by 443
- 2019InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximizationcited by 441
- 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
- 2015Deep Learning with Limited Numerical Precisioncited by 368
- 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 326
- 2021Graph Contrastive Learning Automatedcited by 287
- 2019Noise2Self: Blind Denoising by Self-Supervisioncited by 274
- 2017graph2vec: Learning Distributed Representations of Graphscited by 228
- 2020Data Augmentation for Graph Neural Networkscited by 200
- 2022Finding Global Homophily in Graph Neural Networks When Meeting Heterophilycited by 179
- 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
- 2020A unifying mutual information view of metric learning: cross-entropy vs. pairwise lossescited by 100
- 2020APAN: Asynchronous Propagation Attention Network for Real-time Temporal Graph Embeddingcited by 78
- 2021Graph-MLP: Node Classification without Message Passing in Graphcited by 73
- 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Frameworkcited by 69
- 2021Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoodscited by 57
- 2020Pre-Training Graph Neural Networks for Cold-Start Users and Items Representationcited by 42
- 2021Accelerating Large Scale Real-Time GNN Inference using Channel Pruningcited by 41
- 2022Linkless Link Prediction via Relational Distillationcited by 39
- 2023Simple and Asymmetric Graph Contrastive Learning without Augmentationscited by 37
50 most cited of 54 All 54 (JSON)
Built on by
Not cited yet.