# Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Language

arXiv: https://arxiv.org/abs/2303.03363 · HTML: https://mlsanity.com/paper/2303.03363/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 155 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 30 of 57)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [Molecular De Novo Design through Deep Reinforcement Learning](https://mlsanity.com/paper/1704.07555/) (2017)
- [Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development](https://mlsanity.com/paper/2102.09548/) (2021)

## Built on by (most cited first, 30 of 44)

- [Large Language Models on Graphs: A Comprehensive Survey](https://mlsanity.com/paper/2312.02783/) (2023)
- [MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter](https://mlsanity.com/paper/2310.12798/) (2023)
- [Graph Foundation Models: Concepts, Opportunities and Challenges](https://mlsanity.com/paper/2310.11829/) (2023)
- [Scientific Large Language Models: A Survey on Biological & Chemical Domains](https://mlsanity.com/paper/2401.14656/) (2024)
- [GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning](https://mlsanity.com/paper/2306.13089/) (2023)
- [ChemReasoner: Heuristic Search over a Large Language Model's Knowledge Space using Quantum-Chemical Feedback](https://mlsanity.com/paper/2402.10980/) (2024)
- [MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction](https://mlsanity.com/paper/2406.12950/) (2024)
- [L+M-24: Building a Dataset for Language + Molecules @ ACL 2024](https://mlsanity.com/paper/2403.00791/) (2024)
- [Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Design](https://mlsanity.com/paper/2310.14420/) (2023)
- [Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey](https://mlsanity.com/paper/2403.01528/) (2024)
- [ChemMLLM: Chemical Multimodal Large Language Model](https://mlsanity.com/paper/2505.16326/) (2025)
- [Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning](https://mlsanity.com/paper/2508.06199/) (2025)
- [SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design](https://mlsanity.com/paper/2307.11694/) (2023)
- [Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences](https://mlsanity.com/paper/2411.04165/) (2024)
- [Learning Multi-view Molecular Representations with Structured and Unstructured Knowledge](https://mlsanity.com/paper/2406.09841/) (2024)
- [MixMin: Finding Data Mixtures via Convex Minimization](https://mlsanity.com/paper/2502.10510/) (2025)
- [3D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modeling](https://mlsanity.com/paper/2406.05797/) (2024)
- [3M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecular Structure Generation](https://mlsanity.com/paper/2403.07179/) (2024)
- [Bi-level Contrastive Learning for Knowledge-Enhanced Molecule Representations](https://mlsanity.com/paper/2306.01631/) (2023)
- [MolTailor: Tailoring Chemical Molecular Representation to Specific Tasks via Text Prompts](https://mlsanity.com/paper/2401.11403/) (2024)
- [Entropy-Reinforced Planning with Large Language Models for Drug Discovery](https://mlsanity.com/paper/2406.07025/) (2024)
- [Tabular foundation models for in-context prediction of molecular properties](https://mlsanity.com/paper/2604.16123/) (2026)
- [Bridging Text and Molecule: A Survey on Multimodal Frameworks for Molecule](https://mlsanity.com/paper/2403.13830/) (2024)
- [Bridging Text and Crystal Structures: Literature-driven Contrastive Learning for Materials Science](https://mlsanity.com/paper/2501.12919/) (2025)
- [TwinBooster: Synergising Large Language Models with Barlow Twins and Gradient Boosting for Enhanced Molecular Property Prediction](https://mlsanity.com/paper/2401.04478/) (2024)
- [Vision Language Model is NOT All You Need: Augmentation Strategies for Molecule Language Models](https://mlsanity.com/paper/2407.09043/) (2024)
- [AgentDrug: Utilizing Large Language Models in An Agentic Workflow for Zero-Shot Molecular Editing](https://mlsanity.com/paper/2410.13147/) (2024)
- [Computational modeling of semantic change](https://mlsanity.com/paper/2304.06337/) (2023)
- [Instruction Multi-Constraint Molecular Generation Using a Teacher-Student Large Language Model](https://mlsanity.com/paper/2403.13244/) (2024)
- [Unlocking Potential Binders: Multimodal Pretraining DEL-Fusion for Denoising DNA-Encoded Libraries](https://mlsanity.com/paper/2409.05916/) (2024)

## Abstract (arXiv)

Activity and property prediction models are the central workhorses in drug discovery and materials sciences, but currently they have to be trained or fine-tuned for new tasks. Without training or fine-tuning, scientific language models could be used for such low-data tasks through their announced zero- and few-shot capabilities. However, their predictive quality at activity prediction is lacking. In this work, we envision a novel type of activity prediction model that is able to adapt to new prediction tasks at inference time, via understanding textual information describing the task. To this end, we propose a new architecture with separate modules for chemical and natural language inputs, and a contrastive pre-training objective on data from large biochemical databases. In extensive experiments, we show that our method CLAMP yields improved predictive performance on few-shot learning benchmarks and zero-shot problems in drug discovery. We attribute the advances of our method to the modularized architecture and to our pre-training objective.