# SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery

arXiv: https://arxiv.org/abs/1911.04738 · HTML: https://mlsanity.com/paper/1911.04738/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2019.

## Built on by (most cited first, 30 of 90)

- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language](https://mlsanity.com/paper/2209.05481/) (2022)
- [Geometric Deep Learning on Molecular Representations](https://mlsanity.com/paper/2107.12375/) (2021)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective](https://mlsanity.com/paper/2306.06615/) (2023)
- [Scientific Large Language Models: A Survey on Biological & Chemical Domains](https://mlsanity.com/paper/2401.14656/) (2024)
- [GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning](https://mlsanity.com/paper/2306.13089/) (2023)
- [MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design](https://mlsanity.com/paper/2203.14500/) (2022)
- [Large AI Models in Health Informatics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2303.11568/) (2023)
- [Distributed Deep Learning in Open Collaborations](https://mlsanity.com/paper/2106.10207/) (2021)
- [Unified 2D and 3D Pre-Training of Molecular Representations](https://mlsanity.com/paper/2207.08806/) (2022)
- [Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphs](https://mlsanity.com/paper/2110.01191/) (2021)
- [Can Large Language Models Empower Molecular Property Prediction?](https://mlsanity.com/paper/2307.07443/) (2023)
- [TransPolymer: a Transformer-based language model for polymer property predictions](https://mlsanity.com/paper/2209.01307/) (2022)
- [Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets](https://mlsanity.com/paper/2310.04292/) (2023)
- [Large Language Models for Scientific Synthesis, Inference and Explanation](https://mlsanity.com/paper/2310.07984/) (2023)
- [Chemical-Reaction-Aware Molecule Representation Learning](https://mlsanity.com/paper/2109.09888/) (2021)
- [A Systematic Survey of Chemical Pre-trained Models](https://mlsanity.com/paper/2210.16484/) (2022)
- [Fractional Denoising for 3D Molecular Pre-training](https://mlsanity.com/paper/2307.10683/) (2023)
- [Artificial Intelligence in Drug Discovery: Applications and Techniques](https://mlsanity.com/paper/2106.05386/) (2021)
- [Dual-view Molecule Pre-training](https://mlsanity.com/paper/2106.10234/) (2021)
- [Bayesian Optimization of Catalysis With In-Context Learning](https://mlsanity.com/paper/2304.05341/) (2023)
- [Relative Molecule Self-Attention Transformer](https://mlsanity.com/paper/2110.05841/) (2021)
- [Deep Molecular Representation Learning via Fusing Physical and Chemical Information](https://mlsanity.com/paper/2112.04624/) (2021)
- [Learning Attributed Graph Representations with Communicative Message Passing Transformer](https://mlsanity.com/paper/2107.08773/) (2021)
- [textttMiniMol: A Parameter-Efficient Foundation Model for Molecular Learning](https://mlsanity.com/paper/2404.14986/) (2024)
- [Federated Learning of Molecular Properties with Graph Neural Networks in a Heterogeneous Setting](https://mlsanity.com/paper/2109.07258/) (2021)
- [A smile is all you need: Predicting limiting activity coefficients from SMILES with natural language processing](https://mlsanity.com/paper/2206.07048/) (2022)
- [LLM and GNN are Complementary: Distilling LLM for Multimodal Graph Learning](https://mlsanity.com/paper/2406.01032/) (2024)
- [ChemDFM-X: Towards Large Multimodal Model for Chemistry](https://mlsanity.com/paper/2409.13194/) (2024)

## Abstract (arXiv)

In drug-discovery-related tasks such as virtual screening, machine learning is emerging as a promising way to predict molecular properties. Conventionally, molecular fingerprints (numerical representations of molecules) are calculated through rule-based algorithms that map molecules to a sparse discrete space. However, these algorithms perform poorly for shallow prediction models or small datasets. To address this issue, we present SMILES Transformer. Inspired by Transformer and pre-trained language models from natural language processing, SMILES Transformer learns molecular fingerprints through unsupervised pre-training of the sequence-to-sequence language model using a huge corpus of SMILES, a text representation system for molecules. We performed benchmarks on 10 datasets against existing fingerprints and graph-based methods and demonstrated the superiority of the proposed algorithms in small-data settings where pre-training facilitated good generalization. Moreover, we define a novel metric to concurrently measure model accuracy and data efficiency.