There is more to graphs than meets the eye: Learning universal features with self-supervision
Not on the map. Why?
Abstract (arXiv)
We study the problem of learning features through self-supervision that are generalisable to multiple graphs. State-of-the-art graph self-supervision restricts training to only one graph, resulting in graph-specific models that are incompatible with different but related graphs. We hypothesize that training with more than one graph that belong to the same family can improve the quality of the learnt representations. However, learning universal features from disparate node/edge features in different graphs is non-trivial. To address this challenge, we first homogenise the disparate features with graph-specific encoders that transform the features into a common space. A universal representation learning module then learns generalisable features on this common space. We show that compared to traditional self-supervision with one graph, our approach results in (1) better performance on downstream node classification, (2) learning features that can be re-used for unseen graphs of the same family, (3) more efficient training and (4) compact yet generalisable models. We also show ability of the proposed framework to deliver these benefits for relatively larger graphs. In this paper, we present a principled way to design foundation graph models that learn from more than one graph in an end-to-end manner, while bridging the gap between self-supervised and supervised performance.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 34 of its 42 references that are in the map. First posted on arXiv 31 May 2023.
Builds on
34 of 42 references.
References · 34
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
- 2017Graph Attention Networkscited by 8,854
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2017Inductive Representation Learning on Large Graphscited by 6,284
- 2018How Powerful are Graph Neural Networks?cited by 4,355
- 2018Deep Graph Infomaxcited by 1,101
- 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 878
- 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
- 2019Graph Neural Networks for Social Recommendationcited by 711
- 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
- 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
- 2020Heterogeneous Graph Transformercited by 581
22 more
- 2019Graph Transformer Networkscited by 520
- 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
- 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
- 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
- 2020Self-supervised Graph Learning for Recommendationcited by 305
- 2021Graph Self-Supervised Learning: A Surveycited by 289
- 2019Revisiting Self-Supervised Visual Representation Learningcited by 250
- 2023A Cookbook of Self-Supervised Learningcited by 248
- 2021Self-Supervised Learning of Graph Neural Networks: A Unified Reviewcited by 184
- 2021Motif-based Graph Self-Supervised Learning for Molecular Property Predictioncited by 159
- 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
- 2022NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphscited by 137
- 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
- 2021Combinatorial Optimization with Physics-Inspired Graph Neural Networkscited by 107
- 2020Molecular graph generation with Graph Neural Networkscited by 79
- 2021Automated Self-Supervised Learning for Graphscited by 50
- 2020Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Textcited by 46
- 2020Learning to Extrapolate Knowledge: Transductive Few-shot Out-of-Graph Link Predictioncited by 46
- 2021Transformers Generalize DeepSets and Can be Extended to Graphs and Hypergraphscited by 35
- 2020A Graph Neural Network Framework for Causal Inference in Brain Networkscited by 16
- 2022Multimodal foundation models are better simulators of the human braincited by 10
- 2023Tokenized Graph Transformer with Neighborhood Augmentation for Node Classification in Large Graphscited by 4
Built on by
Citing papers per year: 2024: 1.
1 paper.