# OmniGenBench: Automating Large-scale in-silico Benchmarking for Genomic Foundation Models

arXiv: https://arxiv.org/abs/2410.01784 · HTML: https://mlsanity.com/paper/2410.01784/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 69 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://mlsanity.com/paper/2306.15794/) (2023)
- [DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome](https://mlsanity.com/paper/2306.15006/) (2023)
- [Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling](https://mlsanity.com/paper/2403.03234/) (2024)
- [PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding](https://mlsanity.com/paper/2206.02096/) (2022)
- [Deep Learning in Bioinformatics](https://mlsanity.com/paper/1603.06430/) (2016)
- [RNA Secondary Structure Prediction By Learning Unrolled Algorithms](https://mlsanity.com/paper/2002.05810/) (2020)
- [A 5' UTR Language Model for Decoding Untranslated Regions of mRNA and Function Predictions](https://mlsanity.com/paper/2310.03281/) (2023)
- [VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling](https://mlsanity.com/paper/2405.10812/) (2024)
- [GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models](https://mlsanity.com/paper/2406.01627/) (2024)
- [BEACON: Benchmark for Comprehensive RNA Tasks and Language Models](https://mlsanity.com/paper/2406.10391/) (2024)
- [Bridging Sequence-Structure Alignment in RNA Foundation Models](https://mlsanity.com/paper/2407.11242/) (2024)

## Abstract (arXiv)

The advancements in artificial intelligence in recent years, such as Large Language Models (LLMs), have fueled expectations for breakthroughs in genomic foundation models (GFMs). The code of nature, hidden in diverse genomes since the very beginning of life's evolution, holds immense potential for impacting humans and ecosystems through genome modeling. Recent breakthroughs in GFMs, such as Evo, have attracted significant investment and attention to genomic modeling, as they address long-standing challenges and transform in-silico genomic studies into automated, reliable, and efficient paradigms. In the context of this flourishing era of consecutive technological revolutions in genomics, GFM studies face two major challenges: the lack of GFM benchmarking tools and the absence of open-source software for diverse genomics. These challenges hinder the rapid evolution of GFMs and their wide application in tasks such as understanding and synthesizing genomes, problems that have persisted for decades. To address these challenges, we introduce GFMBench, a framework dedicated to GFM-oriented benchmarking. GFMBench standardizes benchmark suites and automates benchmarking for a wide range of open-source GFMs. It integrates millions of genomic sequences across hundreds of genomic tasks from four large-scale benchmarks, democratizing GFMs for a wide range of in-silico genomic applications. Additionally, GFMBench is released as open-source software, offering user-friendly interfaces and diverse tutorials, applicable for AutoBench and complex tasks like RNA design and structure prediction. To facilitate further advancements in genome modeling, we have launched a public leaderboard showcasing the benchmark performance derived from AutoBench. GFMBench represents a step toward standardizing GFM benchmarking and democratizing GFM applications.