# Does GNN Pretraining Help Molecular Representation?

arXiv: https://arxiv.org/abs/2207.06010 · HTML: https://mlsanity.com/paper/2207.06010/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 13 Jul 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Contrastive Multi-View Representation Learning on Graphs](https://mlsanity.com/paper/2006.05582/) (2020)
- [Graph Contrastive Learning with Adaptive Augmentation](https://mlsanity.com/paper/2010.14945/) (2020)
- [Representation Learning on Graphs: Methods and Applications](https://mlsanity.com/paper/1709.05584/) (2017)
- [Do Transformers Really Perform Bad for Graph Representation?](https://mlsanity.com/paper/2106.05234/) (2021)
- [GPT-GNN: Generative Pre-Training of Graph Neural Networks](https://mlsanity.com/paper/2006.15437/) (2020)
- [Towards Deeper Graph Neural Networks](https://mlsanity.com/paper/2007.09296/) (2020)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [Structure-Aware Transformer for Graph Representation Learning](https://mlsanity.com/paper/2202.03036/) (2022)
- [Pure Transformers are Powerful Graph Learners](https://mlsanity.com/paper/2207.02505/) (2022)
- [Self-Supervised Learning of Graph Neural Networks: A Unified Review](https://mlsanity.com/paper/2102.10757/) (2021)
- [SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation](https://mlsanity.com/paper/2202.03104/) (2022)
- [When Does Self-Supervision Help Graph Convolutional Networks?](https://mlsanity.com/paper/2006.09136/) (2020)
- [MoCL: Data-driven Molecular Fingerprint via Knowledge-aware Contrastive Learning from Molecular Graph](https://mlsanity.com/paper/2106.04509/) (2021)
- [Alchemy: A Quantum Chemistry Dataset for Benchmarking AI Models](https://mlsanity.com/paper/1906.09427/) (2019)

## Built on by (most cited first, 30 of 60)

- [GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning](https://mlsanity.com/paper/2306.13089/) (2023)
- [Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Language](https://mlsanity.com/paper/2303.03363/) (2023)
- [Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets](https://mlsanity.com/paper/2310.04292/) (2023)
- [Fast Graph Condensation with Structure-based Neural Tangent Kernel](https://mlsanity.com/paper/2310.11046/) (2023)
- [A Systematic Survey of Chemical Pre-trained Models](https://mlsanity.com/paper/2210.16484/) (2022)
- [Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications](https://mlsanity.com/paper/2306.02592/) (2023)
- [Graph Positional and Structural Encoder](https://mlsanity.com/paper/2307.07107/) (2023)
- [Automated 3D Pre-Training for Molecular Property Prediction](https://mlsanity.com/paper/2306.07812/) (2023)
- [Data-Centric Learning from Unlabeled Graphs with Diffusion Model](https://mlsanity.com/paper/2303.10108/) (2023)
- [FedSSP: Federated Graph Learning with Spectral Knowledge and Personalized Preference](https://mlsanity.com/paper/2410.20105/) (2024)
- [Search to Fine-tune Pre-trained Graph Neural Networks for Graph-level Tasks](https://mlsanity.com/paper/2308.06960/) (2023)
- [Towards Neural Scaling Laws on Graphs](https://mlsanity.com/paper/2402.02054/) (2024)
- [MolCPT: Molecule Continuous Prompt Tuning to Generalize Molecular Representation Learning](https://mlsanity.com/paper/2212.10614/) (2022)
- [Tensor-view Topological Graph Neural Network](https://mlsanity.com/paper/2401.12007/) (2024)
- [Transfer learning for chemically accurate interatomic neural network potentials](https://mlsanity.com/paper/2212.03916/) (2022)
- [Identifying Semantic Component for Robust Molecular Property Prediction](https://mlsanity.com/paper/2311.04837/) (2023)
- [CEGA: A Cost-Effective Approach for Graph-Based Model Extraction and Acquisition](https://mlsanity.com/paper/2506.17709/) (2025)
- [Instructor-inspired Machine Learning for Robust Molecular Property Prediction](https://mlsanity.com/paper/2304.03906/) (2023)
- [Learning Molecular Representation in a Cell](https://mlsanity.com/paper/2406.12056/) (2024)
- [MAGE: Model-Level Graph Neural Networks Explanations via Motif-based Graph Generation](https://mlsanity.com/paper/2405.12519/) (2024)
- [When is an Embedding Model More Promising than Another?](https://mlsanity.com/paper/2406.07640/) (2024)
- [Molecular Topological Profile (MOLTOP) -- Simple and Strong Baseline for Molecular Graph Classification](https://mlsanity.com/paper/2407.12136/) (2024)
- [Knowledge-augmented Graph Machine Learning for Drug Discovery: A Survey](https://mlsanity.com/paper/2302.08261/) (2023)
- [Multi-channel learning for integrating structural hierarchies into context-dependent molecular representation](https://mlsanity.com/paper/2311.02798/) (2023)
- [Improving Self-supervised Molecular Representation Learning using Persistent Homology](https://mlsanity.com/paper/2311.17327/) (2023)
- [Towards Graph Foundation Models: A Study on the Generalization of Positional and Structural Encodings](https://mlsanity.com/paper/2412.07407/) (2024)
- [GraphBridge: Towards Arbitrary Transfer Learning in GNNs](https://mlsanity.com/paper/2502.19252/) (2025)
- [Learning to Group Auxiliary Datasets for Molecule](https://mlsanity.com/paper/2307.04052/) (2023)
- [Unifying Invariance and Spuriousity for Graph Out-of-Distribution via Probability of Necessity and Sufficiency](https://mlsanity.com/paper/2402.09165/) (2024)
- [Omni-Mol: Multitask Molecular Model for Any-to-any Modalities](https://mlsanity.com/paper/2502.01074/) (2025)

## Abstract (arXiv)

Extracting informative representations of molecules using Graph neural networks (GNNs) is crucial in AI-driven drug discovery. Recently, the graph research community has been trying to replicate the success of self-supervised pretraining in natural language processing, with several successes claimed. However, we find the benefit brought by self-supervised pretraining on small molecular data can be negligible in many cases. We conduct thorough ablation studies on the key components of GNN pretraining, including pretraining objectives, data splitting methods, input features, pretraining dataset scales, and GNN architectures, to see how they affect the accuracy of the downstream tasks. Our first important finding is, self-supervised graph pretraining do not always have statistically significant advantages over non-pretraining methods in many settings. Secondly, although noticeable improvement can be observed with additional supervised pretraining, the improvement may diminish with richer features or more balanced data splits. Thirdly, hyper-parameters could have larger impacts on accuracy of downstream tasks than the choice of pretraining tasks, especially when the scales of downstream tasks are small. Finally, we provide our conjectures where the complexity of some pretraining methods on small molecules might be insufficient, followed by empirical evidences on different pretraining datasets.