# Molecular representation learning with language models and domain-relevant auxiliary tasks

arXiv: https://arxiv.org/abs/2011.13230 · HTML: https://mlsanity.com/paper/2011.13230/

## Key facts

- Cited by 94 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2020.

## Built on by (most cited first, 30 of 94)

- [Translation between Molecules and Natural Language](https://mlsanity.com/paper/2204.11817/) (2022)
- [ChemBERTa-2: Towards Chemical Foundation Models](https://mlsanity.com/paper/2209.01712/) (2022)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Large AI Models in Health Informatics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2303.11568/) (2023)
- [Regression Transformer: Concurrent sequence regression and generation for molecular language modeling](https://mlsanity.com/paper/2202.01338/) (2022)
- [Quantifying the Gain in Weak-to-Strong Generalization](https://mlsanity.com/paper/2405.15116/) (2024)
- [SELFormer: Molecular Representation Learning via SELFIES Language Models](https://mlsanity.com/paper/2304.04662/) (2023)
- [Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast](https://mlsanity.com/paper/2202.09346/) (2022)
- [Chemical-Reaction-Aware Molecule Representation Learning](https://mlsanity.com/paper/2109.09888/) (2021)
- [ChemReasoner: Heuristic Search over a Large Language Model's Knowledge Space using Quantum-Chemical Feedback](https://mlsanity.com/paper/2402.10980/) (2024)
- [Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformers](https://mlsanity.com/paper/2307.14367/) (2023)
- [Artificial Intelligence in Drug Discovery: Applications and Techniques](https://mlsanity.com/paper/2106.05386/) (2021)
- [Dual-view Molecule Pre-training](https://mlsanity.com/paper/2106.10234/) (2021)
- [nach0: Multimodal Natural and Chemical Languages Foundation Model](https://mlsanity.com/paper/2311.12410/) (2023)
- [BioBridge: Bridging Biomedical Foundation Models via Knowledge Graphs](https://mlsanity.com/paper/2310.03320/) (2023)
- [ReLM: Leveraging Language Models for Enhanced Chemical Reaction Prediction](https://mlsanity.com/paper/2310.13590/) (2023)
- [MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension](https://mlsanity.com/paper/2403.08192/) (2024)
- [Relative Molecule Self-Attention Transformer](https://mlsanity.com/paper/2110.05841/) (2021)
- [A graph representation of molecular ensembles for polymer property prediction](https://mlsanity.com/paper/2205.08619/) (2022)
- [Searching for High-Value Molecules Using Reinforcement Learning and Transformers](https://mlsanity.com/paper/2310.02902/) (2023)
- [Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Design](https://mlsanity.com/paper/2310.14420/) (2023)
- [Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Model](https://mlsanity.com/paper/2502.13449/) (2025)
- [Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning](https://mlsanity.com/paper/2508.06199/) (2025)
- [GPT-MolBERTa: GPT Molecular Features Language Model for molecular property prediction](https://mlsanity.com/paper/2310.03030/) (2023)
- [SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design](https://mlsanity.com/paper/2307.11694/) (2023)
- [MolE: a molecular foundation model for drug discovery](https://mlsanity.com/paper/2211.02657/) (2022)
- [Exploiting Pretrained Biochemical Language Models for Targeted Drug Design](https://mlsanity.com/paper/2209.00981/) (2022)
- [Otter-Knowledge: benchmarks of multimodal knowledge graph representation learning from different sources for drug discovery](https://mlsanity.com/paper/2306.12802/) (2023)
- [Can Large Language Models Understand Molecules?](https://mlsanity.com/paper/2402.00024/) (2024)

## Abstract (arXiv)

We apply a Transformer architecture, specifically BERT, to learn flexible and high quality molecular representations for drug discovery problems. We study the impact of using different combinations of self-supervised tasks for pre-training, and present our results for the established Virtual Screening and QSAR benchmarks. We show that: i) The selection of appropriate self-supervised task(s) for pre-training has a significant impact on performance in subsequent downstream tasks such as Virtual Screening. ii) Using auxiliary tasks with more domain relevance for Chemistry, such as learning to predict calculated molecular properties, increases the fidelity of our learnt representations. iii) Finally, we show that molecular representations learnt by our model `MolBert' improve upon the current state of the art on the benchmark datasets.