# BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations

arXiv: https://arxiv.org/abs/2310.07276 · HTML: https://mlsanity.com/paper/2310.07276/

## Key facts

- Cited by 100 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 101 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction](https://mlsanity.com/paper/2106.06130/) (2021)
- [Fr'echet ChemNet Distance: A metric for generative models for molecules in drug discovery](https://mlsanity.com/paper/1803.09518/) (2018)
- [Translation between Molecules and Natural Language](https://mlsanity.com/paper/2204.11817/) (2022)
- [Motif-based Graph Self-Supervised Learning for Molecular Property Prediction](https://mlsanity.com/paper/2110.00987/) (2021)
- [A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language](https://mlsanity.com/paper/2209.05481/) (2022)
- [Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective](https://mlsanity.com/paper/2306.06615/) (2023)
- [MolXPT: Wrapping Molecules with Text for Generative Pre-training](https://mlsanity.com/paper/2305.10688/) (2023)
- [ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Texts](https://mlsanity.com/paper/2301.12040/) (2023)
- [PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding](https://mlsanity.com/paper/2206.02096/) (2022)
- [SELFIES and the future of molecular string representations](https://mlsanity.com/paper/2204.00056/) (2022)
- [MolTrans: Molecular Interaction Transformer for Drug Target Interaction Prediction](https://mlsanity.com/paper/2004.11424/) (2020)
- [A Text-guided Protein Design Framework](https://mlsanity.com/paper/2302.04611/) (2023)

## Built on by (most cited first, 30 of 100)

- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Advancing Multimodal Medical Capabilities of Gemini](https://mlsanity.com/paper/2405.03162/) (2024)
- [Scientific Large Language Models: A Survey on Biological & Chemical Domains](https://mlsanity.com/paper/2401.14656/) (2024)
- [Efficient Evolutionary Search Over Chemical Space with Large Language Models](https://mlsanity.com/paper/2406.16976/) (2024)
- [BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning](https://mlsanity.com/paper/2402.17810/) (2024)
- [TxGemma: Efficient and Agentic LLMs for Therapeutics](https://mlsanity.com/paper/2504.06196/) (2025)
- [LICO: Large Language Models for In-Context Molecular Optimization](https://mlsanity.com/paper/2406.18851/) (2024)
- [Tx-LLM: A Large Language Model for Therapeutics](https://mlsanity.com/paper/2406.06316/) (2024)
- [Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generation](https://mlsanity.com/paper/2412.14642/) (2024)
- [MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction](https://mlsanity.com/paper/2406.12950/) (2024)
- [MolTC: Towards Molecular Relational Modeling In Language Models](https://mlsanity.com/paper/2402.03781/) (2024)
- [MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension](https://mlsanity.com/paper/2403.08192/) (2024)
- [L+M-24: Building a Dataset for Language + Molecules @ ACL 2024](https://mlsanity.com/paper/2403.00791/) (2024)
- [ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training](https://mlsanity.com/paper/2403.07920/) (2024)
- [A quantitative analysis of knowledge-learning preferences in large language models in molecular science](https://mlsanity.com/paper/2402.04119/) (2024)
- [Large Language Models are In-Context Molecule Learners](https://mlsanity.com/paper/2403.04197/) (2024)
- [Protein Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2502.17504/) (2025)
- [RAG-Enhanced Collaborative LLM Agents for Drug Discovery](https://mlsanity.com/paper/2502.17506/) (2025)
- [Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey](https://mlsanity.com/paper/2403.01528/) (2024)
- [Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Model](https://mlsanity.com/paper/2502.13449/) (2025)
- [ChemMLLM: Chemical Multimodal Large Language Model](https://mlsanity.com/paper/2505.16326/) (2025)
- [From Words to Molecules: A Survey of Large Language Models in Chemistry](https://mlsanity.com/paper/2402.01439/) (2024)
- [PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomes](https://mlsanity.com/paper/2406.13193/) (2024)
- [SmileyLlama: Modifying Large Language Models for Directed Chemical Space Exploration](https://mlsanity.com/paper/2409.02231/) (2024)
- [LDMol: A Text-to-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models](https://mlsanity.com/paper/2405.17829/) (2024)
- [Learning Multi-view Molecular Representations with Structured and Unstructured Knowledge](https://mlsanity.com/paper/2406.09841/) (2024)
- [GP-GPT: Large Language Model for Gene-Phenotype Mapping](https://mlsanity.com/paper/2409.09825/) (2024)
- [MolReFlect: Towards In-Context Fine-grained Alignments between Molecules and Texts](https://mlsanity.com/paper/2411.14721/) (2024)
- [SciReasoner: Laying the Scientific Reasoning Ground Across Disciplines](https://mlsanity.com/paper/2509.21320/) (2025)
- [POLO: Preference-Guided Multi-Turn Reinforcement Learning for Lead Optimization](https://mlsanity.com/paper/2509.21737/) (2025)

## Abstract (arXiv)

Recent advancements in biological research leverage the integration of molecules, proteins, and natural language to enhance drug discovery. However, current models exhibit several limitations, such as the generation of invalid molecular SMILES, underutilization of contextual information, and equal treatment of structured and unstructured knowledge. To address these issues, we propose BioT5, a comprehensive pre-training framework that enriches cross-modal integration in biology with chemical knowledge and natural language associations. BioT5 utilizes SELFIES for 100% robust molecular representations and extracts knowledge from the surrounding context of bio-entities in unstructured biological literature. Furthermore, BioT5 distinguishes between structured and unstructured knowledge, leading to more effective utilization of information. After fine-tuning, BioT5 shows superior performance across a wide range of tasks, demonstrating its strong capability of capturing underlying relations and properties of bio-entities. Our code is available at hrefhttps://github.com/QizhiPei/BioT5Github.