# Multilingual Molecular Representation Learning via Contrastive Pre-training

arXiv: https://arxiv.org/abs/2109.08830 · HTML: https://mlsanity.com/paper/2109.08830/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 60 references that are in the map.
- First posted on arXiv 18 Sep 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery](https://mlsanity.com/paper/1911.04738/) (2019)
- [Mapping the Space of Chemical Reactions Using Attention-Based Neural Networks](https://mlsanity.com/paper/2012.06051/) (2020)
- [Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos](https://mlsanity.com/paper/2104.12671/) (2021)
- [Parameter Efficient Multimodal Transformers for Video Representation Learning](https://mlsanity.com/paper/2012.04124/) (2020)
- [P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding](https://mlsanity.com/paper/2012.13089/) (2020)
- [SMILES2Vec: An Interpretable General-Purpose Deep Neural Network for Predicting Chemical Properties](https://mlsanity.com/paper/1712.02034/) (2017)
- [Dual-view Molecule Pre-training](https://mlsanity.com/paper/2106.10234/) (2021)
- [Contrastive Visual-Linguistic Pretraining](https://mlsanity.com/paper/2007.13135/) (2020)
- [Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning](https://mlsanity.com/paper/2010.08200/) (2020)
- [Latent Molecular Optimization for Targeted Therapeutic Design](https://mlsanity.com/paper/1809.02032/) (2018)
- [BERT Learns (and Teaches) Chemistry](https://mlsanity.com/paper/2007.16012/) (2020)

## Built on by (most cited first, 12 of 12)

- [Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Language](https://mlsanity.com/paper/2303.03363/) (2023)
- [A Systematic Survey of Chemical Pre-trained Models](https://mlsanity.com/paper/2210.16484/) (2022)
- [Fractional Denoising for 3D Molecular Pre-training](https://mlsanity.com/paper/2307.10683/) (2023)
- [From Words to Molecules: A Survey of Large Language Models in Chemistry](https://mlsanity.com/paper/2402.01439/) (2024)
- [Pre-training with Fractional Denoising to Enhance Molecular Property Prediction](https://mlsanity.com/paper/2407.11086/) (2024)
- [ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular Modeling](https://mlsanity.com/paper/2403.12995/) (2024)
- [PolyCL: Contrastive Learning for Polymer Representation Learning via Explicit and Implicit Augmentations](https://mlsanity.com/paper/2408.07556/) (2024)
- [Multilingual and Multimodal Topic Modelling with Pretrained Embeddings](https://mlsanity.com/paper/2211.08057/) (2022)
- [UniMAP: Universal SMILES-Graph Representation Learning](https://mlsanity.com/paper/2310.14216/) (2023)
- [MolTailor: Tailoring Chemical Molecular Representation to Specific Tasks via Text Prompts](https://mlsanity.com/paper/2401.11403/) (2024)
- [MaskMol: Knowledge-guided Molecular Image Pre-Training Framework for Activity Cliffs](https://mlsanity.com/paper/2409.12926/) (2024)
- [Dual-Modality Representation Learning for Molecular Property Prediction](https://mlsanity.com/paper/2501.06608/) (2025)

## Abstract (arXiv)

Molecular representation learning plays an essential role in cheminformatics. Recently, language model-based approaches have gained popularity as an alternative to traditional expert-designed features to encode molecules. However, these approaches only utilize a single molecular language for representation learning. Motivated by the fact that a given molecule can be described using different languages such as Simplified Molecular Line Entry System (SMILES), The International Union of Pure and Applied Chemistry (IUPAC), and The IUPAC International Chemical Identifier (InChI), we propose a multilingual molecular embedding generation approach called MM-Deacon (multilingual molecular domain embedding analysis via contrastive learning). MM-Deacon is pre-trained using SMILES and IUPAC as two different languages on large-scale molecules. We evaluated the robustness of our method on seven molecular property prediction tasks from MoleculeNet benchmark, zero-shot cross-lingual retrieval, and a drug-drug interaction prediction task.