# Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching

arXiv: https://arxiv.org/abs/2206.13602 · HTML: https://mlsanity.com/paper/2206.13602/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 80 references that are in the map.
- First posted on arXiv 27 Jun 2022.

## Builds on (most cited first, 30 of 52)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [E(n) Equivariant Graph Neural Networks](https://mlsanity.com/paper/2102.09844/) (2021)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization](https://mlsanity.com/paper/1908.01000/) (2019)
- [Equivariant message passing for the prediction of tensorial properties and molecular spectra](https://mlsanity.com/paper/2102.03150/) (2021)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [GemNet: Universal Directional Graph Neural Networks for Molecules](https://mlsanity.com/paper/2106.08903/) (2021)
- [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation](https://mlsanity.com/paper/2001.09382/) (2020)
- [GPT-GNN: Generative Pre-Training of Graph Neural Networks](https://mlsanity.com/paper/2006.15437/) (2020)
- [GuacaMol: Benchmarking Models for De Novo Molecular Design](https://mlsanity.com/paper/1811.09621/) (2018)
- [GFlowNet Foundations](https://mlsanity.com/paper/2111.09266/) (2021)
- [Graph Self-Supervised Learning: A Survey](https://mlsanity.com/paper/2103.00111/) (2021)
- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)

## Built on by (most cited first, 30 of 68)

- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editing](https://mlsanity.com/paper/2212.10789/) (2022)
- [GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning](https://mlsanity.com/paper/2306.13089/) (2023)
- [A Text-guided Protein Design Framework](https://mlsanity.com/paper/2302.04611/) (2023)
- [Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets](https://mlsanity.com/paper/2310.04292/) (2023)
- [From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Prediction](https://mlsanity.com/paper/2310.16802/) (2023)
- [A Systematic Survey of Chemical Pre-trained Models](https://mlsanity.com/paper/2210.16484/) (2022)
- [Fractional Denoising for 3D Molecular Pre-training](https://mlsanity.com/paper/2307.10683/) (2023)
- [A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining](https://mlsanity.com/paper/2305.18407/) (2023)
- [A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applications](https://mlsanity.com/paper/2403.00485/) (2024)
- [DiffPack: A Torsional Diffusion Model for Autoregressive Protein Side-Chain Packing](https://mlsanity.com/paper/2306.01794/) (2023)
- [Symmetry-Informed Geometric Representation for Molecules, Proteins, and Crystalline Materials](https://mlsanity.com/paper/2306.09375/) (2023)
- [Graph Neural Networks for Molecules](https://mlsanity.com/paper/2209.05582/) (2022)
- [Few-shot Image Generation with Diffusion Models](https://mlsanity.com/paper/2211.03264/) (2022)
- [Multimodal Molecular Pretraining via Modality Blending](https://mlsanity.com/paper/2307.06235/) (2023)
- [Sliced Denoising: A Physics-Informed Molecular Pre-Training Method](https://mlsanity.com/paper/2311.02124/) (2023)
- [Generalizing Denoising to Non-Equilibrium Structures Improves Equivariant Force Fields](https://mlsanity.com/paper/2403.09549/) (2024)
- [UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learning](https://mlsanity.com/paper/2405.10343/) (2024)
- [Geometry-Complete Perceptron Networks for 3D Molecular Graphs](https://mlsanity.com/paper/2211.02504/) (2022)
- [Evaluating representation learning on the protein structure universe](https://mlsanity.com/paper/2406.13864/) (2024)
- [DomainStudio: Fine-Tuning Diffusion Models for Domain-Driven Image Generation using Limited Data](https://mlsanity.com/paper/2306.14153/) (2023)
- [Protein-ligand binding representation learning from fine-grained interactions](https://mlsanity.com/paper/2311.16160/) (2023)
- [Pre-training with Fractional Denoising to Enhance Molecular Property Prediction](https://mlsanity.com/paper/2407.11086/) (2024)
- [Pre-Training Protein Encoder via Siamese Sequence-Structure Diffusion Trajectory Prediction](https://mlsanity.com/paper/2301.12068/) (2023)
- [ProFSA: Self-supervised Pocket Pretraining via Protein Fragment-Surroundings Alignment](https://mlsanity.com/paper/2310.07229/) (2023)
- [Geometric Representation Condition Improves Equivariant Molecule Generation](https://mlsanity.com/paper/2410.03655/) (2024)
- [Triplet Interaction Improves Graph Transformers: Accurate Molecular Graph Learning with Triplet Graph Transformers](https://mlsanity.com/paper/2402.04538/) (2024)
- [Denoise Pretraining on Nonequilibrium Molecules for Accurate and Transferable Neural Potentials](https://mlsanity.com/paper/2303.02216/) (2023)
- [Molecule Joint Auto-Encoding: Trajectory Pretraining with 2D and 3D Diffusion](https://mlsanity.com/paper/2312.03475/) (2023)
- [Molecular relaxation by reverse diffusion with time step prediction](https://mlsanity.com/paper/2404.10935/) (2024)

## Abstract (arXiv)

Molecular representation pretraining is critical in various applications for drug and material discovery due to the limited number of labeled molecules, and most existing work focuses on pretraining on 2D molecular graphs. However, the power of pretraining on 3D geometric structures has been less explored. This is owing to the difficulty of finding a sufficient proxy task that can empower the pretraining to effectively extract essential features from the geometric structures. Motivated by the dynamic nature of 3D molecules, where the continuous motion of a molecule in the 3D Euclidean space forms a smooth potential energy surface, we propose GeoSSL, a 3D coordinate denoising pretraining framework to model such an energy landscape. Further by leveraging an SE(3)-invariant score matching method, we propose GeoSSL-DDM in which the coordinate denoising proxy task is effectively boiled down to denoising the pairwise atomic distances in a molecule. Our comprehensive experiments confirm the effectiveness and robustness of our proposed method.