# GENEB: Why Genomic Models Are Hard to Compare

arXiv: https://arxiv.org/abs/2606.04525 · HTML: https://mlsanity.com/paper/2606.04525/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 47 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 20 of 20)

- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://mlsanity.com/paper/2306.15794/) (2023)
- [DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome](https://mlsanity.com/paper/2306.15006/) (2023)
- [Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling](https://mlsanity.com/paper/2403.03234/) (2024)
- [BEND: Benchmarking DNA Language Models on biologically meaningful tasks](https://mlsanity.com/paper/2311.12570/) (2023)
- [GENERator: A Long-Context Generative Genomic Foundation Model](https://mlsanity.com/paper/2502.07272/) (2025)
- [VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling](https://mlsanity.com/paper/2405.10812/) (2024)
- [Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA](https://mlsanity.com/paper/2412.13716/) (2024)
- [HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Model](https://mlsanity.com/paper/2502.10807/) (2025)
- [JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model](https://mlsanity.com/paper/2505.17257/) (2025)
- [SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model](https://mlsanity.com/paper/2506.01833/) (2025)
- [BMFM-DNA: A SNP-aware DNA foundation model to capture variant effects](https://mlsanity.com/paper/2507.05265/) (2025)
- [Omni-DNA: A Unified Genomic Foundation Model for Cross-Modal and Multi-Task Learning](https://mlsanity.com/paper/2502.03499/) (2025)
- [Gene42: Long-Range Genomic Foundation Model With Dense Attention](https://mlsanity.com/paper/2503.16565/) (2025)
- [OmniGenBench: A Modular Platform for Reproducible Genomic Foundation Models Benchmarking](https://mlsanity.com/paper/2505.14402/) (2025)
- [OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks](https://mlsanity.com/paper/2505.18775/) (2025)
- [HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling](https://mlsanity.com/paper/2505.20836/) (2025)
- [DNAGPT: A Generalized Pre-trained Tool for Versatile DNA Sequence Analysis Tasks](https://mlsanity.com/paper/2307.05628/) (2023)
- [DNABERT-S: Pioneering Species Differentiation with Species-Aware DNA Embeddings](https://mlsanity.com/paper/2402.08777/) (2024)
- [eccDNAMamba: A Pre-Trained Model for Ultra-Long eccDNA Sequence Analysis](https://mlsanity.com/paper/2506.18940/) (2025)

## Abstract (arXiv)

Progress in genomic foundation models is difficult to assess due to fragmented benchmarks, incompatible evaluation protocols, and task-specific reporting. As a result, claims of superiority or generality across models are often not directly comparable. We introduce GENEB, a large-scale diagnostic benchmark that evaluates frozen representations from 40 genomic foundation models across 100 tasks spanning 13 functional categories under a unified probing-based protocol, including few-shot regimes. GENEB enables controlled comparison across model scale, architecture, tokenization, and pretraining data while explicitly exposing task-level trade-offs. Our analysis shows that aggregate leaderboards are unstable: model rankings vary sharply across task categories, scale provides only modest and inconsistent gains, and architectural and pretraining alignment frequently outweigh parameter count. These results highlight limitations of current evaluation practices and position GENEB as a reference framework for principled comparison and category-aware model selection in genomic machine learning.