# TheBioCollection: Unified Pre-Training Scale LLM Corpus for Biology

arXiv: https://arxiv.org/abs/2607.08803 · HTML: https://mlsanity.com/paper/2607.08803/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 96 references that are in the map.
- First posted on arXiv 9 Jul 2026.

## Builds on (most cited first, 27 of 27)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development](https://mlsanity.com/paper/2102.09548/) (2021)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [ProGen2: Exploring the Boundaries of Protein Language Models](https://mlsanity.com/paper/2206.13517/) (2022)
- [Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models](https://mlsanity.com/paper/2306.08018/) (2023)
- [LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset](https://mlsanity.com/paper/2402.09391/) (2024)
- [BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations](https://mlsanity.com/paper/2310.07276/) (2023)
- [PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding](https://mlsanity.com/paper/2206.02096/) (2022)
- [BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning](https://mlsanity.com/paper/2402.17810/) (2024)
- [TxGemma: Efficient and Agentic LLMs for Therapeutics](https://mlsanity.com/paper/2504.06196/) (2025)
- [Nature Language Model: Deciphering the Language of Nature for Scientific Discovery](https://mlsanity.com/paper/2502.07527/) (2025)
- [PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis](https://mlsanity.com/paper/2408.10609/) (2024)
- [BEND: Benchmarking DNA Language Models on biologically meaningful tasks](https://mlsanity.com/paper/2311.12570/) (2023)
- [Tx-LLM: A Large Language Model for Therapeutics](https://mlsanity.com/paper/2406.06316/) (2024)
- [L+M-24: Building a Dataset for Language + Molecules @ ACL 2024](https://mlsanity.com/paper/2403.00791/) (2024)
- [A Fine-tuning Dataset and Benchmark for Large Language Models for Protein Understanding](https://mlsanity.com/paper/2406.05540/) (2024)
- [A quantitative analysis of knowledge-learning preferences in large language models in molecular science](https://mlsanity.com/paper/2402.04119/) (2024)
- [Learning to design protein-protein interactions with enhanced generalization](https://mlsanity.com/paper/2310.18515/) (2023)
- [SciReasoner: Laying the Scientific Reasoning Ground Across Disciplines](https://mlsanity.com/paper/2509.21320/) (2025)
- [DIPS-Plus: The Enhanced Database of Interacting Protein Structures for Interface Prediction](https://mlsanity.com/paper/2106.04362/) (2021)
- [MolLangBench: A Comprehensive Benchmark for Language-Prompted Molecular Structure Recognition, Editing, and Generation](https://mlsanity.com/paper/2505.15054/) (2025)
- [Speaking the Language of Science: Toward a General-Purpose Generative Foundation Model for the Natural Sciences](https://mlsanity.com/paper/2606.16905/) (2026)
- [Towards Autonomous Mechanistic Reasoning in Virtual Cells](https://mlsanity.com/paper/2604.11661/) (2026)

## Abstract (arXiv)

The push toward large language models for biology (BioLM) has created a need for training corpora that can endow models with a genuine understanding of biology. However, existing biological resources, such as molecular databases, protein repositories, genomic annotations, single-cell atlases, and pathway databases, are scattered across heterogeneous formats and remain unorganized into a cohesive corpus for language model training. We present TheBioCollection, a 52.6B-token pre-training-scale corpus that converts these disparate resources into a unified, training-ready form spanning small molecules, proteins, genomic sequences, cells, and pathways. Beyond consolidating existing data, TheBioCollection enriches each record with tool-computed biological properties and introduces new instruction tasks for capabilities that current corpora barely cover. We pair the corpus with TheBioCollection-Eval, a matched suite probing recognition, generation, and prediction across molecular, protein, genomic, cellular, and cross-domain settings. Holding the base Gravity-16B-A3B architecture fixed, training on TheBioCollection more than doubles its overall score on TheBioCollection-Eval with gains in every domain, while leaving general linguistic ability nearly intact.