# Exploiting Pretrained Biochemical Language Models for Targeted Drug Design

arXiv: https://arxiv.org/abs/2209.00981 · HTML: https://mlsanity.com/paper/2209.00981/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 52 references that are in the map.
- First posted on arXiv 2 Sep 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models](https://mlsanity.com/paper/1811.12823/) (2018)
- [Fr'echet ChemNet Distance: A metric for generative models for molecules in drug discovery](https://mlsanity.com/paper/1803.09518/) (2018)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation](https://mlsanity.com/paper/2005.10283/) (2020)
- [Molecular representation learning with language models and domain-relevant auxiliary tasks](https://mlsanity.com/paper/2011.13230/) (2020)
- [Variational Transformer Networks for Layout Generation](https://mlsanity.com/paper/2104.02416/) (2021)
- [Variational Transformers for Diverse Response Generation](https://mlsanity.com/paper/2003.12738/) (2020)
- [CogMol: Target-Specific and Selective Drug Design for COVID-19 Using Deep Generative Models](https://mlsanity.com/paper/2004.01215/) (2020)
- [Latent Molecular Optimization for Targeted Therapeutic Design](https://mlsanity.com/paper/1809.02032/) (2018)
- [Pre-training Protein Language Models with Label-Agnostic Binding Pairs Enhances Performance in Downstream Tasks](https://mlsanity.com/paper/2012.03084/) (2020)

## Built on by (most cited first, 10 of 10)

- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [Scientific Large Language Models: A Survey on Biological & Chemical Domains](https://mlsanity.com/paper/2401.14656/) (2024)
- [Advancing biomolecular understanding and design following human instructions](https://mlsanity.com/paper/2410.07919/) (2024)
- [Target Specific De Novo Design of Drug Candidate Molecules with Graph Transformer-based Generative Adversarial Networks](https://mlsanity.com/paper/2302.07868/) (2023)
- [Structure-based drug discovery with deep learning](https://mlsanity.com/paper/2212.13295/) (2022)
- [The Role of Language Models in Modern Healthcare: A Comprehensive Review](https://mlsanity.com/paper/2409.16860/) (2024)
- [ActivityDiff: A diffusion model with Positive and Negative Activity Guidance for De Novo Drug Design](https://mlsanity.com/paper/2508.06364/) (2025)
- [Advances in Large Language Models for Medicine](https://mlsanity.com/paper/2509.18690/) (2025)
- [A Hitchhiker's Guide to Deep Chemical Language Processing for Bioactivity Prediction](https://mlsanity.com/paper/2407.12152/) (2024)
- [Generative Artificial Intelligence in Bioinformatics: A Systematic Review of Models, Applications, and Methodological Advances](https://mlsanity.com/paper/2511.03354/) (2025)

## Abstract (arXiv)

Motivation: The development of novel compounds targeting proteins of interest is one of the most important tasks in the pharmaceutical industry. Deep generative models have been applied to targeted molecular design and have shown promising results. Recently, target-specific molecule generation has been viewed as a translation between the protein language and the chemical language. However, such a model is limited by the availability of interacting protein-ligand pairs. On the other hand, large amounts of unlabeled protein sequences and chemical compounds are available and have been used to train language models that learn useful representations. In this study, we propose exploiting pretrained biochemical language models to initialize (i.e. warm start) targeted molecule generation models. We investigate two warm start strategies: (i) a one-stage strategy where the initialized model is trained on targeted molecule generation (ii) a two-stage strategy containing a pre-finetuning on molecular generation followed by target specific training. We also compare two decoding strategies to generate compounds: beam search and sampling. Results: The results show that the warm-started models perform better than a baseline model trained from scratch. The two proposed warm-start strategies achieve similar results to each other with respect to widely used metrics from benchmarks. However, docking evaluation of the generated compounds for a number of novel proteins suggests that the one-stage strategy generalizes better than the two-stage strategy. Additionally, we observe that beam search outperforms sampling in both docking evaluation and benchmark metrics for assessing compound quality. Availability and implementation: The source code is available at https://github.com/boun-tabi/biochemical-lms-for-drug-design and the materials are archived in Zenodo at https://doi.org/10.5281/zenodo.6832145