# Dual-view Molecule Pre-training

arXiv: https://arxiv.org/abs/2106.10234 · HTML: https://mlsanity.com/paper/2106.10234/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 63 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Contrastive Multi-View Representation Learning on Graphs](https://mlsanity.com/paper/2006.05582/) (2020)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [DeeperGCN: All You Need to Train Deeper GCNs](https://mlsanity.com/paper/2006.07739/) (2020)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [Retrosynthesis Prediction with Conditional Graph Logic Network](https://mlsanity.com/paper/2001.01408/) (2020)
- [Molecular representation learning with language models and domain-relevant auxiliary tasks](https://mlsanity.com/paper/2011.13230/) (2020)
- [N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Molecules](https://mlsanity.com/paper/1806.09206/) (2018)
- [Molecule Attention Transformer](https://mlsanity.com/paper/2002.08264/) (2020)
- [SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery](https://mlsanity.com/paper/1911.04738/) (2019)
- [Retrosynthetic reaction prediction using neural sequence-to-sequence models](https://mlsanity.com/paper/1706.01643/) (2017)
- [A Graph to Graphs Framework for Retrosynthesis Prediction](https://mlsanity.com/paper/2003.12725/) (2020)

## Built on by (most cited first, 27 of 27)

- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [3D Infomax improves GNNs for Molecular Property Prediction](https://mlsanity.com/paper/2110.04126/) (2021)
- [ChemBERTa-2: Towards Chemical Foundation Models](https://mlsanity.com/paper/2209.01712/) (2022)
- [The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4](https://mlsanity.com/paper/2311.07361/) (2023)
- [Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction](https://mlsanity.com/paper/2110.09681/) (2021)
- [Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast](https://mlsanity.com/paper/2202.09346/) (2022)
- [Crystal Twins: Self-supervised Learning for Crystalline Material Property Prediction](https://mlsanity.com/paper/2205.01893/) (2022)
- [A Systematic Survey of Chemical Pre-trained Models](https://mlsanity.com/paper/2210.16484/) (2022)
- [Fractional Denoising for 3D Molecular Pre-training](https://mlsanity.com/paper/2307.10683/) (2023)
- [Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Model](https://mlsanity.com/paper/2211.10590/) (2022)
- [A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applications](https://mlsanity.com/paper/2202.07893/) (2022)
- [Graph Neural Networks for Molecules](https://mlsanity.com/paper/2209.05582/) (2022)
- [ChemDFM-X: Towards Large Multimodal Model for Chemistry](https://mlsanity.com/paper/2409.13194/) (2024)
- [Multilingual Molecular Representation Learning via Contrastive Pre-training](https://mlsanity.com/paper/2109.08830/) (2021)
- [Retroformer: Pushing the Limits of Interpretable End-to-end Retrosynthesis Transformer](https://mlsanity.com/paper/2201.12475/) (2022)
- [SSM-DTA: Breaking the Barriers of Data Scarcity in Drug-Target Affinity Prediction](https://mlsanity.com/paper/2206.09818/) (2022)
- [UniMAP: Universal SMILES-Graph Representation Learning](https://mlsanity.com/paper/2310.14216/) (2023)
- [Modeling Diverse Chemical Reactions for Single-step Retrosynthesis via Discrete Latent Variables](https://mlsanity.com/paper/2208.05482/) (2022)
- [Substructure-Atom Cross Attention for Molecular Representation Learning](https://mlsanity.com/paper/2210.08243/) (2022)
- [Pre-trained Molecular Language Models with Random Functional Group Masking](https://mlsanity.com/paper/2411.01401/) (2024)
- [MIN: Multi-channel Interaction Network for Drug-Target Interaction with Protein Distillation](https://mlsanity.com/paper/2412.07778/) (2024)
- [Are VAEs Bad at Reconstructing Molecular Graphs?](https://mlsanity.com/paper/2305.03041/) (2023)
- [Large-scale Pretraining Improves Sample Efficiency of Active Learning based Molecule Virtual Screening](https://mlsanity.com/paper/2309.11687/) (2023)
- [Dual-Modality Representation Learning for Molecular Property Prediction](https://mlsanity.com/paper/2501.06608/) (2025)
- [Molecular Property Prediction by Semantic-invariant Contrastive Learning](https://mlsanity.com/paper/2303.06902/) (2023)
- [ADMEOOD: Out-of-Distribution Benchmark for Drug Property Prediction](https://mlsanity.com/paper/2310.07253/) (2023)
- [MolFusion: Multimodal Fusion Learning for Molecular Representations via Multi-granularity Views](https://mlsanity.com/paper/2406.18020/) (2024)

## Abstract (arXiv)

Inspired by its success in natural language processing and computer vision, pre-training has attracted substantial attention in cheminformatics and bioinformatics, especially for molecule based tasks. A molecule can be represented by either a graph (where atoms are connected by bonds) or a SMILES sequence (where depth-first-search is applied to the molecular graph with specific rules). Existing works on molecule pre-training use either graph representations only or SMILES representations only. In this work, we propose to leverage both the representations and design a new pre-training algorithm, dual-view molecule pre-training (briefly, DMP), that can effectively combine the strengths of both types of molecule representations. The model of DMP consists of two branches: a Transformer branch that takes the SMILES sequence of a molecule as input, and a GNN branch that takes a molecular graph as input. The training of DMP contains three tasks: (1) predicting masked tokens in a SMILES sequence by the Transformer branch, (2) predicting masked atoms in a molecular graph by the GNN branch, and (3) maximizing the consistency between the two high-level representations output by the Transformer and GNN branches separately. After pre-training, we can use either the Transformer branch (this one is recommended according to empirical results), the GNN branch, or both for downstream tasks. DMP is tested on nine molecular property prediction tasks and achieves state-of-the-art performances on seven of them. Furthermore, we test DMP on three retrosynthesis tasks and achieve state-of-the-art results on them.