mlsanityData to 7 Oct 2026

A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applications

arXiv 2202.07893 ↗First posted 16 Feb 2022cs.LG cs.SI q-bio.BMcited by 23

Not on the map. Why?

Abstract (arXiv)

Pretrained Language Models (PLMs) such as BERT have revolutionized the landscape of Natural Language Processing (NLP). Inspired by their proliferation, tremendous efforts have been devoted to Pretrained Graph Models (PGMs). Owing to the powerful model architectures of PGMs, abundant knowledge from massive labeled and unlabeled graph data can be captured. The knowledge implicitly encoded in model parameters can benefit various downstream tasks and help to alleviate several fundamental issues of learning on graphs. In this paper, we provide the first comprehensive survey for PGMs. We firstly present the limitations of graph representation learning and thus introduce the motivation for graph pre-training. Then, we systematically categorize existing PGMs based on a taxonomy from four different perspectives. Next, we present the applications of PGMs in social recommendation and drug discovery. Finally, we outline several promising research directions that can serve as a guideline for future research.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 54 of its 81 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

54 of 81 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2018How Powerful are Graph Neural Networks?cited by 4,355
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  7. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
  8. 2014DeepWalk: Online Learning of Social Representationscited by 1,543
  9. 2016Variational Graph Auto-Encoderscited by 1,373
  10. 2020Graph Contrastive Learning with Augmentationscited by 1,203
  11. 2018Deep Graph Infomaxcited by 1,101
  12. 2019Strategies for Pre-training Graph Neural Networkscited by 964
38 more
  1. 2019Heterogeneous Graph Attention Networkcited by 732
  2. 2020Contrastive Multi-View Representation Learning on Graphscited by 651
  3. 2015LINE: Large-scale Information Network Embeddingcited by 582
  4. 2020Heterogeneous Graph Transformercited by 581
  5. 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 569
  6. 2020Deep Graph Contrastive Representation Learningcited by 535
  7. 2020Graph Contrastive Learning with Adaptive Augmentationcited by 519
  8. 2020Principal Neighbourhood Aggregation for Graph Netscited by 477
  9. 2020GCC: Graph Contrastive Coding for Graph Neural Network Pre-Trainingcited by 461
  10. 2019InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximizationcited by 441
  11. 2018Session-based Recommendation with Graph Neural Networkscited by 312
  12. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 302
  13. 2021Graph Self-Supervised Learning: A Surveycited by 289
  14. 2021Graph Contrastive Learning Automatedcited by 287
  15. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  16. 2020Graph Representation Learning via Graphical Mutual Information Maximizationcited by 217
  17. 2021ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Predictioncited by 204
  18. 2020Graph-Bert: Only Attention is Needed for Learning Graph Representationscited by 176
  19. 2022SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentationcited by 171
  20. 2021Motif-based Graph Self-Supervised Learning for Molecular Property Predictioncited by 159
  21. 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 138
  22. 2021Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learningcited by 130
  23. 2021Self-supervised Graph-level Representation Learning with Local and Global Structurecited by 125
  24. 2021An Empirical Study of Graph Contrastive Learningcited by 113
  25. 2021Self-Supervised Multi-Channel Hypergraph Convolutional Network for Social Recommendationcited by 101
  26. 2015PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networkscited by 96
  27. 2018N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Moleculescited by 93
  28. 2020Rethinking pooling in graph neural networkscited by 73
  29. 2021Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learningcited by 70
  30. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  31. 2019Semi-Implicit Graph Variational Auto-Encoderscited by 54
  32. 2021Molecular Contrastive Learning with Chemical Element Knowledge Graphcited by 50
  33. 2020Pre-Training Graph Neural Networks for Cold-Start Users and Items Representationcited by 42
  34. 2021Adaptive Transfer Learning on Graph Neural Networkscited by 31
  35. 2020An Efficient Neighborhood-based Interaction Model for Recommendation on Heterogeneous Graphcited by 29
  36. 2022Bringing Your Own View: Graph Contrastive Learning without Prefabricated Data Augmentationscited by 28
  37. 2021Dual-view Molecule Pre-trainingcited by 27
  38. 2020TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processingcited by 25

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 10; 2024: 7; 2025: 0; 2026: 1.

23 papers.

Sort:

  1. 2023GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networkscited by 166
  2. 2022Universal Prompt Tuning for Graph Neural Networkscited by 100
  3. 2022A Survey on Deep Graph Generation: Methods and Applicationscited by 69
  4. 2022MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Designcited by 66
  5. 2023Generalized Graph Prompt: Toward a Unification of Pre-Training and Downstream Tasks on Graphscited by 39
  6. 2024Continual Learning on Graphs: Challenges, Solutions, and Opportunitiescited by 20
  7. 2023Motif-aware Attribute Masking for Molecular Graph Pre-trainingcited by 16
  8. 2024Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?cited by 13
  9. 2022Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddingscited by 11
  10. 2023BeMap: Balanced Message Passing for Fair Graph Neural Networkcited by 11
  11. 2024A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspectivecited by 9
  12. 2023Self-Pro: A Self-Prompt and Tuning Framework for Graph Neural Networkscited by 8
11 more
  1. 2023PSP: Pre-Training and Structure Prompt Tuning for Graph Neural Networkscited by 8
  2. 2024Where to Mask: Structure-Guided Masking for Graph Masked Autoencoderscited by 7
  3. 2024Beyond Generalization: A Survey of Out-Of-Distribution Adaptation on Graphscited by 6
  4. 2023Joint Pre-training and Local Re-training: Transferable Representation Learning on Multi-source Knowledge Graphscited by 5
  5. 2024A Survey of Few-Shot Learning on Graphs: from Meta-Learning to Pre-Training and Prompt Learningcited by 5
  6. 2024Label Informed Contrastive Pretraining for Node Importance Estimation on Knowledge Graphscited by 5
  7. 2022BatmanNet: Bi-branch Masked Graph Transformer Autoencoder for Molecular Representationcited by 4
  8. 2023When Graph Neural Network Meets Causality: Opportunities, Methodologies and An Outlookcited by 4
  9. 2023Why Deep Models Often cannot Beat Non-deep Counterparts on Molecular Property Prediction?cited by 3
  10. 2023PepLand: a large-scale pre-trained peptide representation model for a comprehensive landscape of both canonical and non-canonical amino acidscited by 0
  11. 2026GraphSculptor: Sculpting Pre-training Coreset for Graph Self-supervised Learningcited by 0