# Unified 2D and 3D Pre-Training of Molecular Representations

arXiv: https://arxiv.org/abs/2207.08806 · HTML: https://mlsanity.com/paper/2207.08806/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 52 references that are in the map.
- First posted on arXiv 14 Jul 2022.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation](https://mlsanity.com/paper/2203.02923/) (2022)
- [Equivariant message passing for the prediction of tensorial properties and molecular spectra](https://mlsanity.com/paper/2102.03150/) (2021)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](https://mlsanity.com/paper/2103.09430/) (2021)
- [Cormorant: Covariant Molecular Neural Networks](https://mlsanity.com/paper/1906.04015/) (2019)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [GEOM: Energy-annotated molecular conformations for property prediction and molecular generation](https://mlsanity.com/paper/2006.05531/) (2020)
- [Learning Gradient Fields for Molecular Conformation Generation](https://mlsanity.com/paper/2105.03902/) (2021)
- [3D Infomax improves GNNs for Molecular Property Prediction](https://mlsanity.com/paper/2110.04126/) (2021)
- [Molecular geometry prediction using a deep generative graph neural network](https://mlsanity.com/paper/1904.00314/) (2019)
- [SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery](https://mlsanity.com/paper/1911.04738/) (2019)
- [One Model To Learn Them All](https://mlsanity.com/paper/1706.05137/) (2017)
- [Molecular Property Prediction: A Multilevel Quantum Interactions Modeling Perspective](https://mlsanity.com/paper/1906.11081/) (2019)
- [A Generative Model for Molecular Distance Geometry](https://mlsanity.com/paper/1909.11459/) (2019)

## Built on by (most cited first, 30 of 45)

- [MiDi: Mixed Graph and 3D Denoising Diffusion for Molecule Generation](https://mlsanity.com/paper/2302.09048/) (2023)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [One Transformer Can Understand Both 2D & 3D Molecular Data](https://mlsanity.com/paper/2210.01765/) (2022)
- [A Comprehensive Survey on Deep Graph Representation Learning](https://mlsanity.com/paper/2304.05055/) (2023)
- [MolFM: A Multimodal Molecular Foundation Model](https://mlsanity.com/paper/2307.09484/) (2023)
- [Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Language](https://mlsanity.com/paper/2303.03363/) (2023)
- [From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Prediction](https://mlsanity.com/paper/2310.16802/) (2023)
- [Energy-Motivated Equivariant Pretraining for 3D Molecular Graphs](https://mlsanity.com/paper/2207.08824/) (2022)
- [A Systematic Survey of Chemical Pre-trained Models](https://mlsanity.com/paper/2210.16484/) (2022)
- [Fractional Denoising for 3D Molecular Pre-training](https://mlsanity.com/paper/2307.10683/) (2023)
- [Automated 3D Pre-Training for Molecular Property Prediction](https://mlsanity.com/paper/2306.07812/) (2023)
- [Uncovering Neural Scaling Laws in Molecular Representation Learning](https://mlsanity.com/paper/2309.15123/) (2023)
- [Multimodal Molecular Pretraining via Modality Blending](https://mlsanity.com/paper/2307.06235/) (2023)
- [Fragment-based Pretraining and Finetuning on Molecular Graphs](https://mlsanity.com/paper/2310.03274/) (2023)
- [Sliced Denoising: A Physics-Informed Molecular Pre-Training Method](https://mlsanity.com/paper/2311.02124/) (2023)
- [Shift-Robust Molecular Relational Learning with Causal Substructure](https://mlsanity.com/paper/2305.18451/) (2023)
- [Pre-training with Fractional Denoising to Enhance Molecular Property Prediction](https://mlsanity.com/paper/2407.11086/) (2024)
- [MMPolymer: A Multimodal Multitask Pretraining Framework for Polymer Property Prediction](https://mlsanity.com/paper/2406.04727/) (2024)
- [Learning Multi-view Molecular Representations with Structured and Unstructured Knowledge](https://mlsanity.com/paper/2406.09841/) (2024)
- [Beyond Efficiency: Molecular Data Pruning for Enhanced Generalization](https://mlsanity.com/paper/2409.01081/) (2024)
- [MolSpectra: Pre-training 3D Molecular Representation with Multi-modal Energy Spectra](https://mlsanity.com/paper/2502.16284/) (2025)
- [Multi-task Bioassay Pre-training for Protein-ligand Binding Affinity Prediction](https://mlsanity.com/paper/2306.04886/) (2023)
- [ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular Modeling](https://mlsanity.com/paper/2403.12995/) (2024)
- [Bi-level Contrastive Learning for Knowledge-Enhanced Molecule Representations](https://mlsanity.com/paper/2306.01631/) (2023)
- [ChiENN: Embracing Molecular Chirality with Graph Neural Networks](https://mlsanity.com/paper/2307.02198/) (2023)
- [A Survey of Few-Shot Learning on Graphs: from Meta-Learning to Pre-Training and Prompt Learning](https://mlsanity.com/paper/2402.01440/) (2024)
- [Impact of Domain Knowledge and Multi-Modality on Intelligent Molecular Property Prediction: A Systematic Survey](https://mlsanity.com/paper/2402.07249/) (2024)
- [3DMolFormer: A Dual-channel Framework for Structure-based Drug Discovery](https://mlsanity.com/paper/2502.05107/) (2025)
- [Geometry-aware Line Graph Transformer Pre-training for Molecular Property Prediction](https://mlsanity.com/paper/2309.00483/) (2023)
- [Multi-Type Point Cloud Autoencoder: A Complete Equivariant Embedding for Molecule Conformation and Pose](https://mlsanity.com/paper/2405.13791/) (2024)

## Abstract (arXiv)

Molecular representation learning has attracted much attention recently. A molecule can be viewed as a 2D graph with nodes/atoms connected by edges/bonds, and can also be represented by a 3D conformation with 3-dimensional coordinates of all atoms. We note that most previous work handles 2D and 3D information separately, while jointly leveraging these two sources may foster a more informative representation. In this work, we explore this appealing idea and propose a new representation learning method based on a unified 2D and 3D pre-training. Atom coordinates and interatomic distances are encoded and then fused with atomic representations through graph neural networks. The model is pre-trained on three tasks: reconstruction of masked atoms and coordinates, 3D conformation generation conditioned on 2D graph, and 2D graph generation conditioned on 3D conformation. We evaluate our method on 11 downstream molecular property prediction tasks: 7 with 2D information only and 4 with both 2D and 3D information. Our method achieves state-of-the-art results on 10 tasks, and the average improvement on 2D-only tasks is 8.3%. Our method also achieves significant improvement on two 3D conformation generation tasks.