mlsanityData to 7 Oct 2026

Strategies for Pre-training Graph Neural Networks

arXiv 1905.12265 ↗First posted 29 May 2019cs.LG stat.MLcited by 964

Not on the map. Why?

Abstract (arXiv)

Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and then fine-tune it on a downstream task of interest. While pre-training has been effective in many language and vision domains, it remains an open question how to effectively use pre-training on graph datasets. In this paper, we develop a new strategy and self-supervised methods for pre-training Graph Neural Networks (GNNs). The key to the success of our strategy is to pre-train an expressive GNN at the level of individual nodes as well as entire graphs so that the GNN can learn useful local and global representations simultaneously. We systematically study pre-training on multiple graph classification datasets. We find that naive strategies, which pre-train GNNs at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer on many downstream tasks. In contrast, our strategy avoids negative transfer and improves generalization significantly across downstream tasks, leading up to 9.4% absolute improvements in ROC-AUC over non-pre-trained models and achieving state-of-the-art performance for molecular property prediction and protein function prediction.

Key facts. Cited by 964 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 110; 2022: 171; 2023: 193; 2024: 221; 2025: 143; 2026: 126.

964 papers.

Sort:

By area:

  1. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  2. 2022GraphMAE: Self-Supervised Masked Graph Autoencoderscited by 443
  3. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  4. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  5. 2021Graph Self-Supervised Learning: A Surveycited by 289
  6. 2021Graph Contrastive Learning Automatedcited by 287
  7. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 258
  8. 2022Long Range Graph Benchmarkcited by 258
  9. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  10. 2022Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistencycited by 246
  11. 2021Weisfeiler and Lehman Go Cellular: CW Networkscited by 218
  12. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
38 more
  1. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  2. 2021ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Predictioncited by 204
  3. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 203
  4. 2021Adversarial Graph Augmentation to Improve Graph Contrastive Learningcited by 200
  5. 2022Structure-Aware Transformer for Graph Representation Learningcited by 191
  6. 2021Large-Scale Representation Learning on Graphs via Bootstrappingcited by 184
  7. 2021Self-Supervised Learning of Graph Neural Networks: A Unified Reviewcited by 184
  8. 2022Sign and Basis Invariant Networks for Spectral Graph Representation Learningcited by 176
  9. 2022SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentationcited by 171
  10. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  11. 2023GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networkscited by 166
  12. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 165
  13. 2021Learning Gradient Fields for Molecular Conformation Generationcited by 162
  14. 2022Protein Representation Learning by Geometric Structure Pretrainingcited by 161
  15. 2021Motif-based Graph Self-Supervised Learning for Molecular Property Predictioncited by 159
  16. 2023Graph Inductive Biases in Transformers without Message Passingcited by 150
  17. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 147
  18. 2023All in One: Multi-task Prompting for Graph Neural Networkscited by 147
  19. 20213D Infomax improves GNNs for Molecular Property Predictioncited by 146
  20. 2021Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Predictioncited by 137
  21. 2021Self-supervised Learning on Graphs: Contrastive, Generative,or Predictivecited by 136
  22. 2021Interpreting and Unifying Graph Neural Networks with An Optimization Frameworkcited by 131
  23. 2023GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learnercited by 131
  24. 2021Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learningcited by 130
  25. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 129
  26. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  27. 2021MARS: Markov Molecular Sampling for Multi-objective Drug Discoverycited by 125
  28. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  29. 2022ChemBERTa-2: Towards Chemical Foundation Modelscited by 123
  30. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 120
  31. 2022A Comprehensive Survey on Trustworthy Graph Neural Networks: Privacy, Robustness, Fairness, and Explainabilitycited by 118
  32. 2023Machine Learning for Synthetic Data Generation: A Reviewcited by 118
  33. 2021DropGNN: Random Dropouts Increase the Expressiveness of Graph Neural Networkscited by 117
  34. 2021An Empirical Study of Graph Contrastive Learningcited by 113
  35. 2021SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networkscited by 108
  36. 2022How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervisioncited by 102
  37. 2022AUC Maximization in the Era of Big Data and AI: A Surveycited by 100
  38. 2022Universal Prompt Tuning for Graph Neural Networkscited by 100

50 most cited of 964 All 964 (JSON)