# GPS++: An Optimised Hybrid MPNN/Transformer for Molecular Property Prediction

arXiv: https://arxiv.org/abs/2212.02229 · HTML: https://mlsanity.com/paper/2212.02229/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 18 Nov 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Recipe for a General, Powerful, Scalable Graph Transformer](https://mlsanity.com/paper/2205.12454/) (2022)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [Rethinking Graph Transformers with Spectral Attention](https://mlsanity.com/paper/2106.03893/) (2021)
- [Graph Neural Networks with Learnable Structural and Positional Representations](https://mlsanity.com/paper/2110.07875/) (2021)
- [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](https://mlsanity.com/paper/2103.09430/) (2021)
- [Pre-training via Denoising for Molecular Property Prediction](https://mlsanity.com/paper/2206.00133/) (2022)
- [Global Self-Attention as a Replacement for Graph Convolution](https://mlsanity.com/paper/2108.03348/) (2021)
- [Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networks](https://mlsanity.com/paper/2203.00199/) (2022)
- [One Transformer Can Understand Both 2D & 3D Molecular Data](https://mlsanity.com/paper/2210.01765/) (2022)
- [GRPE: Relative Positional Encoding for Graph Transformer](https://mlsanity.com/paper/2201.12787/) (2022)
- [Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets](https://mlsanity.com/paper/2203.04810/) (2022)
- [First Place Solution of KDD Cup 2021 & OGB Large-Scale Challenge Graph Prediction Track](https://mlsanity.com/paper/2106.08279/) (2021)
- [Reducing Down(stream)time: Pretraining Molecular GNNs using Heterogeneous AI Accelerators](https://mlsanity.com/paper/2211.04598/) (2022)

## Built on by (most cited first, 28 of 28)

- [Graph Inductive Biases in Transformers without Message Passing](https://mlsanity.com/paper/2305.17589/) (2023)
- [Position: Graph Foundation Models are Already Here](https://mlsanity.com/paper/2402.02216/) (2024)
- [Attending to Graph Transformers](https://mlsanity.com/paper/2302.04181/) (2023)
- [On the Scalability of GNNs for Molecular Graphs](https://mlsanity.com/paper/2404.11568/) (2024)
- [Automated 3D Pre-Training for Molecular Property Prediction](https://mlsanity.com/paper/2306.07812/) (2023)
- [Highly Accurate Quantum Chemical Property Prediction with Uni-Mol+](https://mlsanity.com/paper/2303.16982/) (2023)
- [PubChemQC B3LYP/6-31G*//PM6 dataset: the Electronic Structures of 86 Million Molecules using B3LYP/6-31G* calculations](https://mlsanity.com/paper/2305.18454/) (2023)
- [Triplet Interaction Improves Graph Transformers: Accurate Molecular Graph Learning with Triplet Graph Transformers](https://mlsanity.com/paper/2402.04538/) (2024)
- [Geometry Informed Tokenization of Molecules for Language Model Generation](https://mlsanity.com/paper/2408.10120/) (2024)
- [Platonic Transformers: A Solid Choice For Equivariance](https://mlsanity.com/paper/2510.03511/) (2025)
- [Generalizing Graph Transformers Across Diverse Graphs and Tasks via Pre-training](https://mlsanity.com/paper/2407.03953/) (2024)
- [CellCLIP -- Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learning](https://mlsanity.com/paper/2506.06290/) (2025)
- [Union Subgraph Neural Networks](https://mlsanity.com/paper/2305.15747/) (2023)
- [Impact of Domain Knowledge and Multi-Modality on Intelligent Molecular Property Prediction: A Systematic Survey](https://mlsanity.com/paper/2402.07249/) (2024)
- [GeoMFormer: A General Architecture for Geometric Molecular Representation Learning](https://mlsanity.com/paper/2406.16853/) (2024)
- [GeoTMI:Predicting quantum chemical property with easy-to-obtain geometry via positional denoising](https://mlsanity.com/paper/2304.03724/) (2023)
- [Structural Priors and Modular Adapters in the Composable Fine-Tuning Algorithm of Large-Scale Models](https://mlsanity.com/paper/2511.03981/) (2025)
- [Enhancing Molecular Property Predictions by Learning from Bond Modelling and Interactions](https://mlsanity.com/paper/2603.00568/) (2026)
- [Differentiable Cluster Graph Neural Network](https://mlsanity.com/paper/2405.16185/) (2024)
- [Probabilistic Contrastive Pretraining for Multi-task ADME Property Prediction](https://mlsanity.com/paper/2606.11508/) (2026)
- [A Systematic Survey and Benchmark of Deep Learning for Molecular Property Prediction in the Foundation Model Era](https://mlsanity.com/paper/2604.16586/) (2026)
- [Is Self-Supervised Pretraining Good for Extrapolation in Molecular Property Prediction?](https://mlsanity.com/paper/2308.08129/) (2023)
- [Generating QM1B with PySCFIPU](https://mlsanity.com/paper/2311.01135/) (2023)
- [Efficient Training of Transformers for Molecule Property Prediction on Small-scale Datasets](https://mlsanity.com/paper/2409.04909/) (2024)
- [M3-20M: A Large-Scale Multi-Modal Molecule Dataset for AI-driven Drug Design and Discovery](https://mlsanity.com/paper/2412.06847/) (2024)
- [AdvSynGNN: Structure-Adaptive Graph Neural Nets via Adversarial Synthesis and Self-Corrective Propagation](https://mlsanity.com/paper/2602.17071/) (2026)
- [MolFM-Lite: Multi-Modal Molecular Property Prediction with Conformer Ensemble Attention and Cross-Modal Fusion](https://mlsanity.com/paper/2602.22405/) (2026)
- [k-Maximum Inner Product Attention for Graph Transformers and the Expressive Power of GraphGPS](https://mlsanity.com/paper/2604.03815/) (2026)

## Abstract (arXiv)

This technical report presents GPS++, the first-place solution to the Open Graph Benchmark Large-Scale Challenge (OGB-LSC 2022) for the PCQM4Mv2 molecular property prediction task. Our approach implements several key principles from the prior literature. At its core our GPS++ method is a hybrid MPNN/Transformer model that incorporates 3D atom positions and an auxiliary denoising task. The effectiveness of GPS++ is demonstrated by achieving 0.0719 mean absolute error on the independent test-challenge PCQM4Mv2 split. Thanks to Graphcore IPU acceleration, GPS++ scales to deep architectures (16 layers), training at 3 minutes per epoch, and large ensemble (112 models), completing the final predictions in 1 hour 32 minutes, well under the 4 hour inference budget allocated. Our implementation is publicly available at: https://github.com/graphcore/ogb-lsc-pcqm4mv2.