# Reducing Down(stream)time: Pretraining Molecular GNNs using Heterogeneous AI Accelerators

arXiv: https://arxiv.org/abs/2211.04598 · HTML: https://mlsanity.com/paper/2211.04598/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 8 Nov 2022.

## Builds on (most cited first, 8 of 8)

- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [ChemBERTa-2: Towards Chemical Foundation Models](https://mlsanity.com/paper/2209.01712/) (2022)
- [Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphs](https://mlsanity.com/paper/2110.01191/) (2021)
- [A Survey of FPGA-Based Neural Network Accelerator](https://mlsanity.com/paper/1712.08934/) (2017)
- [HydroNet: Benchmark Tasks for Preserving Intermolecular Interactions and Structural Motifs in Predictive and Generative Models for Molecular Data](https://mlsanity.com/paper/2012.00131/) (2020)
- [Scalable Geometric Deep Learning on Molecular Graphs](https://mlsanity.com/paper/2112.03364/) (2021)

## Built on by (most cited first, 2 of 2)

- [GPS++: An Optimised Hybrid MPNN/Transformer for Molecular Property Prediction](https://mlsanity.com/paper/2212.02229/) (2022)
- [GPS++: Reviving the Art of Message Passing for Molecular Property Prediction](https://mlsanity.com/paper/2302.02947/) (2023)

## Abstract (arXiv)

The demonstrated success of transfer learning has popularized approaches that involve pretraining models from massive data sources and subsequent finetuning towards a specific task. While such approaches have become the norm in fields such as natural language processing, implementation and evaluation of transfer learning approaches for chemistry are in the early stages. In this work, we demonstrate finetuning for downstream tasks on a graph neural network (GNN) trained over a molecular database containing 2.7 million water clusters. The use of Graphcore IPUs as an AI accelerator for training molecular GNNs reduces training time from a reported 2.7 days on 0.5M clusters to 1.2 hours on 2.7M clusters. Finetuning the pretrained model for downstream tasks of molecular dynamics and transfer to a different potential energy surface took only 8.3 hours and 28 minutes, respectively, on a single GPU.