# GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding

arXiv: https://arxiv.org/abs/2604.05774 · HTML: https://mlsanity.com/paper/2604.05774/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 46 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Builds on (most cited first, 9 of 9)

- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset](https://mlsanity.com/paper/2306.03030/) (2023)
- [Benchmarking large language models for biomedical natural language processing applications and recommendations](https://mlsanity.com/paper/2305.16326/) (2023)
- [BioReason: Incentivizing Multimodal Biological Reasoning within a DNA-LLM Model](https://mlsanity.com/paper/2505.23579/) (2025)
- [BEND: Benchmarking DNA Language Models on biologically meaningful tasks](https://mlsanity.com/paper/2311.12570/) (2023)
- [DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA](https://mlsanity.com/paper/2412.05430/) (2024)
- [CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Research](https://mlsanity.com/paper/2510.11985/) (2025)

## Built on by (most cited first, 1 of 1)

- [Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection](https://mlsanity.com/paper/2609.10221/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly adopted as conversational assistants in genomics, where they are mainly used to reason over biological knowledge, annotations, and analysis outputs through natural language interfaces. However, existing benchmarks either focus on specialized DNA models trained for sequence prediction or evaluate biological knowledge using text-only questions, leaving the behavior of general-purpose LLMs when directly exposed to raw genome sequences underexplored. We introduce GenomeQA, a benchmark designed to provide a controlled evaluation setting for general-purpose LLMs on sequence-based genome inference tasks. GenomeQA comprises 5,200 samples drawn from multiple biological databases, with sequence lengths ranging from 6 to 1,000 base pairs (bp), spanning six task families: Enhancer and Promoter Identification, Splice Site Identification, Taxonomic Classification, Histone Mark Prediction, Transcription Factor Binding Site Prediction, and TF Motif Prediction. Across six frontier LLMs, we find that models consistently outperform random baselines and can exploit local sequence signals such as GC content and short motifs, while performance degrades on tasks that require more indirect or multi-step inference over sequence patterns. GenomeQA establishes a diagnostic benchmark for studying and improving the use of general-purpose LLMs on raw genomic sequences.