# KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction

arXiv: https://arxiv.org/abs/2206.03364 · HTML: https://mlsanity.com/paper/2206.03364/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 63 references that are in the map.
- First posted on arXiv 2 Jun 2022.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Principal Neighbourhood Aggregation for Graph Nets](https://mlsanity.com/paper/2004.05718/) (2020)
- [Rethinking Graph Transformers with Spectral Attention](https://mlsanity.com/paper/2106.03893/) (2021)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [Do Transformers Really Perform Bad for Graph Representation?](https://mlsanity.com/paper/2106.05234/) (2021)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [Graph Contrastive Learning Automated](https://mlsanity.com/paper/2106.07594/) (2021)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [GraphiT: Encoding Graph Structure in Transformers](https://mlsanity.com/paper/2106.05667/) (2021)
- [3D Infomax improves GNNs for Molecular Property Prediction](https://mlsanity.com/paper/2110.04126/) (2021)
- [Directional Graph Networks](https://mlsanity.com/paper/2010.02863/) (2020)
- [Self-supervised Graph-level Representation Learning with Local and Global Structure](https://mlsanity.com/paper/2106.04113/) (2021)

## Built on by (most cited first, 30 of 31)

- [SELFormer: Molecular Representation Learning via SELFIES Language Models](https://mlsanity.com/paper/2304.04662/) (2023)
- [A Systematic Survey of Chemical Pre-trained Models](https://mlsanity.com/paper/2210.16484/) (2022)
- [Graph Transformers: A Survey](https://mlsanity.com/paper/2407.09777/) (2024)
- [Structure Pretraining and Prompt Tuning for Knowledge Graph Transfer](https://mlsanity.com/paper/2303.03922/) (2023)
- [A Survey of Graph Transformers: Architectures, Theories and Applications](https://mlsanity.com/paper/2502.16533/) (2025)
- [Identifying Semantic Component for Robust Molecular Property Prediction](https://mlsanity.com/paper/2311.04837/) (2023)
- [Structure Guided Multi-modal Pre-trained Transformer for Knowledge Graph Reasoning](https://mlsanity.com/paper/2307.03591/) (2023)
- [Deep Prompt Tuning for Graph Transformers](https://mlsanity.com/paper/2309.10131/) (2023)
- [Subgraphormer: Unifying Subgraph GNNs and Graph Transformers via Graph Products](https://mlsanity.com/paper/2402.08450/) (2024)
- [Beyond Sequence: Impact of Geometric Context for RNA Property Prediction](https://mlsanity.com/paper/2410.11933/) (2024)
- [Knowledge-augmented Graph Machine Learning for Drug Discovery: A Survey](https://mlsanity.com/paper/2302.08261/) (2023)
- [Impact of Domain Knowledge and Multi-Modality on Intelligent Molecular Property Prediction: A Systematic Survey](https://mlsanity.com/paper/2402.07249/) (2024)
- [BatmanNet: Bi-branch Masked Graph Transformer Autoencoder for Molecular Representation](https://mlsanity.com/paper/2211.13979/) (2022)
- [Uni-QSAR: an Auto-ML Tool for Molecular Property Prediction](https://mlsanity.com/paper/2304.12239/) (2023)
- [Cross-Modal Learning for Chemistry Property Prediction: Large Language Models Meet Graph Machine Learning](https://mlsanity.com/paper/2408.14964/) (2024)
- [Pin-Tuning: Parameter-Efficient In-Context Tuning for Few-Shot Molecular Property Prediction](https://mlsanity.com/paper/2411.01158/) (2024)
- [Aggregate to Adapt: Node-Centric Aggregation for Multi-Source-Free Graph Domain Adaptation](https://mlsanity.com/paper/2502.03033/) (2025)
- [Drug Synergistic Combinations Predictions via Large-Scale Pre-Training and Graph Structure Learning](https://mlsanity.com/paper/2301.05931/) (2023)
- [Knowledge-enhanced Relation Graph and Task Sampling for Few-shot Molecular Property Prediction](https://mlsanity.com/paper/2405.15544/) (2024)
- [Molecular Graph Representation Learning Integrating Large Language Models with Domain-specific Small Models](https://mlsanity.com/paper/2408.10124/) (2024)
- [Scalable Multi-Task Transfer Learning for Molecular Property Prediction](https://mlsanity.com/paper/2410.00432/) (2024)
- [Probabilistic Contrastive Pretraining for Multi-task ADME Property Prediction](https://mlsanity.com/paper/2606.11508/) (2026)
- [Hi-GMAE: Hierarchical Graph Masked Autoencoders](https://mlsanity.com/paper/2405.10642/) (2024)
- [SiMilarity-Enhanced Homophily for Multi-View Heterophilous Graph Clustering](https://mlsanity.com/paper/2410.03596/) (2024)
- [Heterogenous Ensemble of Models for Molecular Property Prediction](https://mlsanity.com/paper/2211.11035/) (2022)
- [Molecular Property Prediction by Semantic-invariant Contrastive Learning](https://mlsanity.com/paper/2303.06902/) (2023)
- [Atomic and Subgraph-aware Bilateral Aggregation for Molecular Representation Learning](https://mlsanity.com/paper/2305.12618/) (2023)
- [3D Denoisers are Good 2D Teachers: Molecular Pretraining via Denoising and Cross-Modal Distillation](https://mlsanity.com/paper/2309.04062/) (2023)
- [MultiPUFFIN: A Multimodal Domain-Constrained Foundation Model for Molecular Property Prediction of Small Molecules](https://mlsanity.com/paper/2603.00857/) (2026)
- [Learning Molecular Representations from Cellular Phenotypes with Structure Preservation](https://mlsanity.com/paper/2608.02688/) (2026)

## Abstract (arXiv)

Designing accurate deep learning models for molecular property prediction plays an increasingly essential role in drug and material discovery. Recently, due to the scarcity of labeled molecules, self-supervised learning methods for learning generalizable and transferable representations of molecular graphs have attracted lots of attention. In this paper, we argue that there exist two major issues hindering current self-supervised learning methods from obtaining desired performance on molecular property prediction, that is, the ill-defined pre-training tasks and the limited model capacity. To this end, we introduce Knowledge-guided Pre-training of Graph Transformer (KPGT), a novel self-supervised learning framework for molecular graph representation learning, to alleviate the aforementioned issues and improve the performance on the downstream molecular property prediction tasks. More specifically, we first introduce a high-capacity model, named Line Graph Transformer (LiGhT), which emphasizes the importance of chemical bonds and is mainly designed to model the structural information of molecular graphs. Then, a knowledge-guided pre-training strategy is proposed to exploit the additional knowledge of molecules to guide the model to capture the abundant structural and semantic information from large-scale unlabeled molecular graphs. Extensive computational tests demonstrated that KPGT can offer superior performance over current state-of-the-art methods on several molecular property prediction tasks.