# Same model, better performance: the impact of shuffling on DNA Language Models benchmarking

arXiv: https://arxiv.org/abs/2510.12617 · HTML: https://mlsanity.com/paper/2510.12617/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 3 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Builds on (most cited first, 3 of 3)

- [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://mlsanity.com/paper/2306.15794/) (2023)
- [BEND: Benchmarking DNA Language Models on biologically meaningful tasks](https://mlsanity.com/paper/2311.12570/) (2023)
- [High Performance I/O For Large Scale Deep Learning](https://mlsanity.com/paper/2001.01858/) (2020)

## Built on by (most cited first, 1 of 1)

- [annbatch unlocks terabyte-scale training of biological data in anndata](https://mlsanity.com/paper/2604.01949/) (2026)

## Abstract (arXiv)

Large Language Models are increasingly popular in genomics due to their potential to decode complex biological sequences. Hence, researchers require a standardized benchmark to evaluate DNA Language Models (DNA LMs) capabilities. However, evaluating DNA LMs is a complex task that intersects genomic's domain-specific challenges and machine learning methodologies, where seemingly minor implementation details can significantly compromise benchmark validity. We demonstrate this through BEND (Benchmarking DNA Language Models), where hardware-dependent hyperparameters -- number of data loading workers and buffer sizes -- create spurious performance variations of up to 4% for identical models. The problem stems from inadequate data shuffling interacting with domain specific data characteristics. Experiments with three DNA language models (HyenaDNA, DNABERT-2, ResNet-LM) show these artifacts affect both absolute performance and relative model rankings. We propose a simple solution: pre-shuffling data before storage eliminates hardware dependencies while maintaining efficiency. This work highlights how standard ML practices can interact unexpectedly with domain-specific data characteristics, with broader implications for benchmark design in specialized domains.