# Self-supervised Learning on Graphs: Contrastive, Generative,or Predictive

arXiv: https://arxiv.org/abs/2105.07342 · HTML: https://mlsanity.com/paper/2105.07342/

## Key facts

- Cited by 136 papers in the map (data through 14 Oct 2026).
- Builds on 82 of its 139 references that are in the map.
- First posted on arXiv 16 May 2021.

## Builds on (most cited first, 30 of 82)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Pitfalls of Graph Neural Network Evaluation](https://mlsanity.com/paper/1811.05868/) (2018)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Contrastive Multi-View Representation Learning on Graphs](https://mlsanity.com/paper/2006.05582/) (2020)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Deep Graph Contrastive Representation Learning](https://mlsanity.com/paper/2006.04131/) (2020)
- [Graph Contrastive Learning with Adaptive Augmentation](https://mlsanity.com/paper/2010.14945/) (2020)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training](https://mlsanity.com/paper/2006.09963/) (2020)
- [InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization](https://mlsanity.com/paper/1908.01000/) (2019)

## Built on by (most cited first, 30 of 136)

- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Data Augmentation for Deep Graph Learning: A Survey](https://mlsanity.com/paper/2202.08235/) (2022)
- [Graph Foundation Models: Concepts, Opportunities and Challenges](https://mlsanity.com/paper/2310.11829/) (2023)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders](https://mlsanity.com/paper/2205.10053/) (2022)
- [OpenGraph: Towards Open Graph Foundation Models](https://mlsanity.com/paper/2403.01121/) (2024)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [Hard Sample Aware Network for Contrastive Deep Graph Clustering](https://mlsanity.com/paper/2212.08665/) (2022)
- [DiffBP: Generative Diffusion of 3D Molecules for Target Protein Binding](https://mlsanity.com/paper/2211.11214/) (2022)
- [OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning](https://mlsanity.com/paper/2306.11249/) (2023)
- [Heterogeneous Graph Masked Autoencoders](https://mlsanity.com/paper/2208.09957/) (2022)
- [Prototypical Graph Contrastive Learning](https://mlsanity.com/paper/2106.09645/) (2021)
- [SGL-PT: A Strong Graph Learner with Graph Prompt Tuning](https://mlsanity.com/paper/2302.12449/) (2023)
- [Self-supervised Heterogeneous Graph Pre-training Based on Structural Clustering](https://mlsanity.com/paper/2210.10462/) (2022)
- [Functional-Group-Based Diffusion for Pocket-Specific Molecule Generation and Elaboration](https://mlsanity.com/paper/2306.13769/) (2023)
- [Uncovering the Structural Fairness in Graph Contrastive Learning](https://mlsanity.com/paper/2210.03011/) (2022)
- [Disentangled Contrastive Learning for Social Recommendation](https://mlsanity.com/paper/2208.08723/) (2022)
- [PPFlow: Target-aware Peptide Design with Torsional Flow Matching](https://mlsanity.com/paper/2405.06642/) (2024)
- [Towards Graph Contrastive Learning: A Survey and Beyond](https://mlsanity.com/paper/2405.11868/) (2024)
- [Automated Spatio-Temporal Graph Contrastive Learning](https://mlsanity.com/paper/2305.03920/) (2023)
- [ADA-GAD: Anomaly-Denoised Autoencoders for Graph Anomaly Detection](https://mlsanity.com/paper/2312.14535/) (2023)
- [Modularity-Aware Graph Autoencoders for Joint Community Detection and Link Prediction](https://mlsanity.com/paper/2202.00961/) (2022)
- [Evaluating Self-Supervised Learning for Molecular Graph Embeddings](https://mlsanity.com/paper/2206.08005/) (2022)
- [Rethinking Graph Masked Autoencoders through Alignment and Uniformity](https://mlsanity.com/paper/2402.07225/) (2024)
- [MAPE-PPI: Towards Effective and Efficient Protein-Protein Interaction Prediction via Microenvironment-Aware Protein Embedding](https://mlsanity.com/paper/2402.14391/) (2024)
- [E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation](https://mlsanity.com/paper/2205.14912/) (2022)
- [Cross-domain recommendation via user interest alignment](https://mlsanity.com/paper/2301.11467/) (2023)
- [Self-supervised Learning for Electroencephalogram: A Systematic Survey](https://mlsanity.com/paper/2401.05446/) (2024)
- [Graph Anomaly Detection in Time Series: A Survey](https://mlsanity.com/paper/2302.00058/) (2023)
- [Towards Reasonable Budget Allocation in Untargeted Graph Structure Attacks via Gradient Debias](https://mlsanity.com/paper/2304.00010/) (2023)

## Abstract (arXiv)

Deep learning on graphs has recently achieved remarkable success on a variety of tasks, while such success relies heavily on the massive and carefully labeled data. However, precise annotations are generally very expensive and time-consuming. To address this problem, self-supervised learning (SSL) is emerging as a new paradigm for extracting informative knowledge through well-designed pretext tasks without relying on manual labels. In this survey, we extend the concept of SSL, which first emerged in the fields of computer vision and natural language processing, to present a timely and comprehensive review of existing SSL techniques for graph data. Specifically, we divide existing graph SSL methods into three categories: contrastive, generative, and predictive. More importantly, unlike other surveys that only provide a high-level description of published research, we present an additional mathematical summary of existing works in a unified framework. Furthermore, to facilitate methodological development and empirical comparisons, we also summarize the commonly used datasets, evaluation metrics, downstream tasks, open-source implementations, and experimental study of various algorithms. Finally, we discuss the technical challenges and potential future directions for improving graph self-supervised learning. Latest advances in graph SSL are summarized in a GitHub repository https://github.com/LirongWu/awesome-graph-self-supervised-learning.