# A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining

arXiv: https://arxiv.org/abs/2305.18407 · HTML: https://mlsanity.com/paper/2305.18407/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 72 references that are in the map.
- First posted on arXiv 28 May 2023.

## Builds on (most cited first, 30 of 48)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [Recipe for a General, Powerful, Scalable Graph Transformer](https://mlsanity.com/paper/2205.12454/) (2022)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020)
- [Principal Neighbourhood Aggregation for Graph Nets](https://mlsanity.com/paper/2004.05718/) (2020)
- [InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization](https://mlsanity.com/paper/1908.01000/) (2019)
- [Graph Self-Supervised Learning: A Survey](https://mlsanity.com/paper/2103.00111/) (2021)
- [Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations](https://mlsanity.com/paper/2202.02514/) (2022)
- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [GEOM: Energy-annotated molecular conformations for property prediction and molecular generation](https://mlsanity.com/paper/2006.05531/) (2020)

## Built on by (most cited first, 30 of 32)

- [A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applications](https://mlsanity.com/paper/2403.00485/) (2024)
- [Symmetry-Informed Geometric Representation for Molecules, Proteins, and Crystalline Materials](https://mlsanity.com/paper/2306.09375/) (2023)
- [Multimodal Molecular Pretraining via Modality Blending](https://mlsanity.com/paper/2307.06235/) (2023)
- [UniGEM: A Unified Approach to Generation and Property Prediction for Molecules](https://mlsanity.com/paper/2410.10516/) (2024)
- [Sliced Denoising: A Physics-Informed Molecular Pre-Training Method](https://mlsanity.com/paper/2311.02124/) (2023)
- [UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learning](https://mlsanity.com/paper/2405.10343/) (2024)
- [MolParser: End-to-end Visual Recognition of Molecule Structures in the Wild](https://mlsanity.com/paper/2411.11098/) (2024)
- [Graph ODEs and Beyond: A Comprehensive Survey on Integrating Differential Equations with Graph Neural Networks](https://mlsanity.com/paper/2503.23167/) (2025)
- [Molecule Joint Auto-Encoding: Trajectory Pretraining with 2D and 3D Diffusion](https://mlsanity.com/paper/2312.03475/) (2023)
- [MolSpectra: Pre-training 3D Molecular Representation with Multi-modal Energy Spectra](https://mlsanity.com/paper/2502.16284/) (2025)
- [CARE: a Benchmark Suite for the Classification and Retrieval of Enzymes](https://mlsanity.com/paper/2406.15669/) (2024)
- [A Multi-Grained Symmetric Differential Equation Model for Learning Protein-Ligand Binding Dynamics](https://mlsanity.com/paper/2401.15122/) (2024)
- [SubGDiff: A Subgraph Diffusion Model to Improve Molecular Representation Learning](https://mlsanity.com/paper/2405.05665/) (2024)
- [SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision](https://mlsanity.com/paper/2412.05569/) (2024)
- [Recent Developments in GNNs for Drug Discovery](https://mlsanity.com/paper/2506.01302/) (2025)
- [GeomCLIP: Contrastive Geometry-Text Pre-training for Molecules](https://mlsanity.com/paper/2411.10821/) (2024)
- [Learning Molecular Chirality via Chiral Determinant Kernels](https://mlsanity.com/paper/2602.07415/) (2026)
- [MolTRES: Improving Chemical Language Representation Learning for Molecular Property Prediction](https://mlsanity.com/paper/2408.01426/) (2024)
- [Manifold-Constrained Nucleus-Level Denoising Diffusion Model for Structure-Based Drug Design](https://mlsanity.com/paper/2409.10584/) (2024)
- [Multi-Modal Molecular Representation Learning via Structure Awareness](https://mlsanity.com/paper/2505.05877/) (2025)
- [MolSight: Molecular Property Prediction with Images](https://mlsanity.com/paper/2605.10157/) (2026)
- [Sculpting Molecules in Text-3D Space: A Flexible Substructure Aware Framework for Text-Oriented Molecular Optimization](https://mlsanity.com/paper/2403.03425/) (2024)
- [3D Interaction Geometric Pre-training for Molecular Relational Learning](https://mlsanity.com/paper/2412.02957/) (2024)
- [M3-20M: A Large-Scale Multi-Modal Molecule Dataset for AI-driven Drug Design and Discovery](https://mlsanity.com/paper/2412.06847/) (2024)
- [A Generalist Cross-Domain Molecular Learning Framework for Structure-Based Drug Discovery](https://mlsanity.com/paper/2503.04362/) (2025)
- [Predicting and generating antibiotics against future pathogens with ApexOracle](https://mlsanity.com/paper/2507.07862/) (2025)
- [Learning the Neighborhood: Contrast-Free Multimodal Self-Supervised Molecular Graph Pretraining](https://mlsanity.com/paper/2509.22468/) (2025)
- [Unified Molecule Pre-training with Flexible 2D and 3D Modalities: Single and Paired Modality Integration](https://mlsanity.com/paper/2510.07035/) (2025)
- [MolGA: Molecular Graph Adaptation with Pre-trained 2D Graph Encoder](https://mlsanity.com/paper/2510.07289/) (2025)
- [Learning 3D Anisotropic Noise Distributions Improves Molecular Force Field Modeling](https://mlsanity.com/paper/2510.22123/) (2025)

## Abstract (arXiv)

Molecule pretraining has quickly become the go-to schema to boost the performance of AI-based drug discovery. Naturally, molecules can be represented as 2D topological graphs or 3D geometric point clouds. Although most existing pertaining methods focus on merely the single modality, recent research has shown that maximizing the mutual information (MI) between such two modalities enhances the molecule representation ability. Meanwhile, existing molecule multi-modal pretraining approaches approximate MI based on the representation space encoded from the topology and geometry, thus resulting in the loss of critical structural information of molecules. To address this issue, we propose MoleculeSDE. MoleculeSDE leverages group symmetric (e.g., SE(3)-equivariant and reflection-antisymmetric) stochastic differential equation models to generate the 3D geometries from 2D topologies, and vice versa, directly in the input space. It not only obtains tighter MI bound but also enables prosperous downstream tasks than the previous work. By comparing with 17 pretraining baselines, we empirically verify that MoleculeSDE can learn an expressive representation with state-of-the-art performance on 26 out of 32 downstream tasks.