# A Fine-tuning Dataset and Benchmark for Large Language Models for Protein Understanding

arXiv: https://arxiv.org/abs/2406.05540 · HTML: https://mlsanity.com/paper/2406.05540/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 8 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024)
- [Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models](https://mlsanity.com/paper/2306.08018/) (2023)
- [ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Texts](https://mlsanity.com/paper/2301.12040/) (2023)
- [PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding](https://mlsanity.com/paper/2206.02096/) (2022)
- [BERN2: an advanced neural biomedical named entity recognition and normalization tool](https://mlsanity.com/paper/2201.02080/) (2022)
- [Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions](https://mlsanity.com/paper/2401.01827/) (2024)
- [Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension](https://mlsanity.com/paper/2112.06494/) (2021)

## Built on by (most cited first, 19 of 19)

- [SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation](https://mlsanity.com/paper/2510.06303/) (2025)
- [Protein Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2502.17504/) (2025)
- [Towards Safeguarding LLM Fine-tuning APIs against Cipher Attacks](https://mlsanity.com/paper/2508.17158/) (2025)
- [TourSynbio: A Multi-Modal Large Model and Agent Framework to Bridge Text and Protein Sequences for Protein Engineering](https://mlsanity.com/paper/2408.15299/) (2024)
- [Detecting Adversarial Fine-tuning with Auditing Agents](https://mlsanity.com/paper/2510.16255/) (2025)
- [Mozi: Governed Autonomy for Drug Discovery LLM Agents](https://mlsanity.com/paper/2603.03655/) (2026)
- [TourSynbio-Search: A Large Language Model Driven Agent Framework for Unified Search Method for Protein Engineering](https://mlsanity.com/paper/2411.06024/) (2024)
- [Rethinking Text-based Protein Understanding: Retrieval or LLM?](https://mlsanity.com/paper/2505.20354/) (2025)
- [SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence](https://mlsanity.com/paper/2512.22334/) (2025)
- [DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learning](https://mlsanity.com/paper/2602.11089/) (2026)
- [Empowering LLMs for Structure-Based Drug Design via Exploration-Augmented Latent Inference](https://mlsanity.com/paper/2601.15333/) (2026)
- [A Survey for Large Language Models in Biomedicine](https://mlsanity.com/paper/2409.00133/) (2024)
- [Evaluating the Effectiveness of Large Language Models in Solving Simple Programming Tasks: A User-Centered Study](https://mlsanity.com/paper/2507.04043/) (2025)
- [ProtRLSearch: A Multi-Round Multimodal Protein Search Agent with Large Language Models Trained via Reinforcement Learning](https://mlsanity.com/paper/2603.01464/) (2026)
- [Protein as a Second Language for LLMs](https://mlsanity.com/paper/2510.11188/) (2025)
- [An Empirical Analysis of Fine-Tuning Large Language Models on Bioinformatics Literature: PRSGPT and BioStarsGPT](https://mlsanity.com/paper/2601.11573/) (2025)
- [MMAI Gym for Science: Training Liquid Foundation Models for Drug Discovery](https://mlsanity.com/paper/2603.03517/) (2026)
- [The limits of bio-molecular modeling with large language models : a cross-scale evaluation](https://mlsanity.com/paper/2604.03361/) (2026)
- [TheBioCollection: Unified Pre-Training Scale LLM Corpus for Biology](https://mlsanity.com/paper/2607.08803/) (2026)

## Abstract (arXiv)

The parallels between protein sequences and natural language in their sequential structures have inspired the application of large language models (LLMs) to protein understanding. Despite the success of LLMs in NLP, their effectiveness in comprehending protein sequences remains an open question, largely due to the absence of datasets linking protein sequences to descriptive text. Researchers have then attempted to adapt LLMs for protein understanding by integrating a protein sequence encoder with a pre-trained LLM. However, this adaptation raises a fundamental question: "Can LLMs, originally designed for NLP, effectively comprehend protein sequences as a form of language?" Current datasets fall short in addressing this question due to the lack of a direct correlation between protein sequences and corresponding text descriptions, limiting the ability to train and evaluate LLMs for protein understanding effectively. To bridge this gap, we introduce ProteinLMDataset, a dataset specifically designed for further self-supervised pretraining and supervised fine-tuning (SFT) of LLMs to enhance their capability for protein sequence comprehension. Specifically, ProteinLMDataset includes 17.46 billion tokens for pretraining and 893,000 instructions for SFT. Additionally, we present ProteinLMBench, the first benchmark dataset consisting of 944 manually verified multiple-choice questions for assessing the protein understanding capabilities of LLMs. ProteinLMBench incorporates protein-related details and sequences in multiple languages, establishing a new standard for evaluating LLMs' abilities in protein comprehension. The large language model InternLM2-7B, pretrained and fine-tuned on the ProteinLMDataset, outperforms GPT-4 on ProteinLMBench, achieving the highest accuracy score.