# Multimodal Molecular Pretraining via Modality Blending

arXiv: https://arxiv.org/abs/2307.06235 · HTML: https://mlsanity.com/paper/2307.06235/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 12 Jul 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [GraphMAE: Self-Supervised Masked Graph Autoencoders](https://mlsanity.com/paper/2205.10803/) (2022)
- [InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization](https://mlsanity.com/paper/1908.01000/) (2019)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](https://mlsanity.com/paper/2103.09430/) (2021)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [Rethinking Positional Encoding in Language Pre-training](https://mlsanity.com/paper/2006.15595/) (2020)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentials](https://mlsanity.com/paper/2202.02541/) (2022)
- [3D Infomax improves GNNs for Molecular Property Prediction](https://mlsanity.com/paper/2110.04126/) (2021)
- [Pre-training via Denoising for Molecular Property Prediction](https://mlsanity.com/paper/2206.00133/) (2022)
- [Revealing the Dark Secrets of Masked Image Modeling](https://mlsanity.com/paper/2205.13543/) (2022)
- [A Mutual Information Maximization Perspective of Language Representation Learning](https://mlsanity.com/paper/1910.08350/) (2019)

## Built on by (most cited first, 21 of 21)

- [MolParser: End-to-end Visual Recognition of Molecule Structures in the Wild](https://mlsanity.com/paper/2411.11098/) (2024)
- [Pre-training with Fractional Denoising to Enhance Molecular Property Prediction](https://mlsanity.com/paper/2407.11086/) (2024)
- [A Survey of Graph Transformers: Architectures, Theories and Applications](https://mlsanity.com/paper/2502.16533/) (2025)
- [MolSpectra: Pre-training 3D Molecular Representation with Multi-modal Energy Spectra](https://mlsanity.com/paper/2502.16284/) (2025)
- [Graph-structured Small Molecule Drug Discovery Through Deep Learning: Progress, Challenges, and Opportunities](https://mlsanity.com/paper/2502.08975/) (2025)
- [Atomas: Hierarchical Alignment on Molecule-Text for Unified Molecule Understanding and Generation](https://mlsanity.com/paper/2404.16880/) (2024)
- [Beyond Atoms: Enhancing Molecular Pretrained Representations with 3D Space Modeling](https://mlsanity.com/paper/2503.10489/) (2025)
- [Bridging the Gap Between Molecule and Textual Descriptions via Substructure-aware Alignment](https://mlsanity.com/paper/2510.26157/) (2025)
- [MolTRES: Improving Chemical Language Representation Learning for Molecular Property Prediction](https://mlsanity.com/paper/2408.01426/) (2024)
- [Learning Cell-Aware Hierarchical Multi-Modal Representations for Robust Molecular Modeling](https://mlsanity.com/paper/2511.21120/) (2025)
- [How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?](https://mlsanity.com/paper/2602.11618/) (2026)
- [MolSight: Molecular Property Prediction with Images](https://mlsanity.com/paper/2605.10157/) (2026)
- [MoleCode unlocks structural intelligence in large language models](https://mlsanity.com/paper/2605.16480/) (2026)
- [MoleculeCLA: Rethinking Molecular Benchmark via Computational Ligand-Target Binding Analysis](https://mlsanity.com/paper/2406.17797/) (2024)
- [M3-20M: A Large-Scale Multi-Modal Molecule Dataset for AI-driven Drug Design and Discovery](https://mlsanity.com/paper/2412.06847/) (2024)
- [Learning the Neighborhood: Contrast-Free Multimodal Self-Supervised Molecular Graph Pretraining](https://mlsanity.com/paper/2509.22468/) (2025)
- [Unified Molecule Pre-training with Flexible 2D and 3D Modalities: Single and Paired Modality Integration](https://mlsanity.com/paper/2510.07035/) (2025)
- [MolGA: Molecular Graph Adaptation with Pre-trained 2D Graph Encoder](https://mlsanity.com/paper/2510.07289/) (2025)
- [Learning 3D Anisotropic Noise Distributions Improves Molecular Force Field Modeling](https://mlsanity.com/paper/2510.22123/) (2025)
- [Retrieval-guided Twin Fusion with Similarity-aware Contrast for Molecule-Text Alignment](https://mlsanity.com/paper/2608.16005/) (2026)
- [MolSC: Leveraging Substituent Contributions to Enhance Fine-grained Molecular Understanding in LLMs](https://mlsanity.com/paper/2609.23073/) (2026)

## Abstract (arXiv)

Self-supervised learning has recently gained growing interest in molecular modeling for scientific tasks such as AI-assisted drug discovery. Current studies consider leveraging both 2D and 3D molecular structures for representation learning. However, relying on straightforward alignment strategies that treat each modality separately, these methods fail to exploit the intrinsic correlation between 2D and 3D representations that reflect the underlying structural characteristics of molecules, and only perform coarse-grained molecule-level alignment. To derive fine-grained alignment and promote structural molecule understanding, we introduce an atomic-relation level "blend-then-predict" self-supervised learning approach, MoleBLEND, which first blends atom relations represented by different modalities into one unified relation matrix for joint encoding, then recovers modality-specific information for 2D and 3D structures individually. By treating atom relationships as anchors, MoleBLEND organically aligns and integrates visually dissimilar 2D and 3D modalities of the same molecule at fine-grained atomic level, painting a more comprehensive depiction of each molecule. Extensive experiments show that MoleBLEND achieves state-of-the-art performance across major 2D/3D molecular benchmarks. We further provide theoretical insights from the perspective of mutual-information maximization, demonstrating that our method unifies contrastive, generative (cross-modality prediction) and mask-then-predict (single-modality prediction) objectives into one single cohesive framework.