# Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting

arXiv: https://arxiv.org/abs/2503.04013 · HTML: https://mlsanity.com/paper/2503.04013/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 80 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Builds on (most cited first, 30 of 40)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [Scientific Large Language Models: A Survey on Biological & Chemical Domains](https://mlsanity.com/paper/2401.14656/) (2024)
- [PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding](https://mlsanity.com/paper/2206.02096/) (2022)
- [ProLLaMA: A Protein Large Language Model for Multi-Task Protein Language Processing](https://mlsanity.com/paper/2402.16445/) (2024)
- [Improving the Robustness of Large Language Models via Consistency Alignment](https://mlsanity.com/paper/2403.14221/) (2024)
- [Robustness of Learning from Task Instructions](https://mlsanity.com/paper/2212.03813/) (2022)
- [InstructProtein: Aligning Human and Protein Language via Knowledge Instruction](https://mlsanity.com/paper/2310.03269/) (2023)
- [UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs](https://mlsanity.com/paper/2404.07584/) (2024)
- [OLAPH: Improving Factuality in Biomedical Long-form Question Answering](https://mlsanity.com/paper/2405.12701/) (2024)

## Built on by (most cited first, 8 of 8)

- [MatTools: Benchmarking Large Language Models for Materials Science Tools](https://mlsanity.com/paper/2505.10852/) (2025)
- [TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agents](https://mlsanity.com/paper/2602.21230/) (2026)
- [SCI-Verifier: Scientific Verifier with Thinking](https://mlsanity.com/paper/2509.24285/) (2025)
- [Domain Specific Benchmarks for Evaluating Multimodal Large Language Models](https://mlsanity.com/paper/2506.12958/) (2025)
- [AssayBench: An Assay-Level Virtual Cell Benchmark for LLMs and Agents](https://mlsanity.com/paper/2605.10876/) (2026)
- [SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models](https://mlsanity.com/paper/2505.15094/) (2025)
- [A Principle-Driven Adaptive Policy for Group Cognitive Stimulation Dialogue for Elderly with Cognitive Impairment](https://mlsanity.com/paper/2603.10034/) (2026)
- [DS-ProGen: A Dual-Structure Deep Language Model for Functional Protein Design](https://mlsanity.com/paper/2505.12511/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have become important tools in solving biological problems, offering improvements in accuracy and adaptability over conventional methods. Several benchmarks have been proposed to evaluate the performance of these LLMs. However, current benchmarks can hardly evaluate the performance of these models across diverse tasks effectively. In this paper, we introduce a comprehensive prompting-based benchmarking framework, termed Bio-benchmark, which includes 30 key bioinformatics tasks covering areas such as proteins, RNA, drugs, electronic health records, and traditional Chinese medicine. Using this benchmark, we evaluate six mainstream LLMs, including GPT-4o and Llama-3.1-70b, etc., using 0-shot and few-shot Chain-of-Thought (CoT) settings without fine-tuning to reveal their intrinsic capabilities. To improve the efficiency of our evaluations, we demonstrate BioFinder, a new tool for extracting answers from LLM responses, which increases extraction accuracy by round 30% compared to existing methods. Our benchmark results show the biological tasks suitable for current LLMs and identify specific areas requiring enhancement. Furthermore, we propose targeted prompt engineering strategies for optimizing LLM performance in these contexts. Based on these findings, we provide recommendations for the development of more robust LLMs tailored for various biological applications. This work offers a comprehensive evaluation framework and robust tools to support the application of LLMs in bioinformatics.