# Genomic Language Models: Opportunities and Challenges

arXiv: https://arxiv.org/abs/2407.11435 · HTML: https://mlsanity.com/paper/2407.11435/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 135 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [A Survey on Neural Network Interpretability](https://mlsanity.com/paper/2012.14261/) (2020)
- [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://mlsanity.com/paper/2306.15794/) (2023)
- [MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers](https://mlsanity.com/paper/2305.07185/) (2023)
- [DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome](https://mlsanity.com/paper/2306.15006/) (2023)
- [Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling](https://mlsanity.com/paper/2403.03234/) (2024)
- [PoET: A generative model of protein families as sequences-of-sequences](https://mlsanity.com/paper/2306.06156/) (2023)
- [BEND: Benchmarking DNA Language Models on biologically meaningful tasks](https://mlsanity.com/paper/2311.12570/) (2023)
- [Are Pre-trained Convolutions Better than Pre-trained Transformers?](https://mlsanity.com/paper/2105.03322/) (2021)
- [A 5' UTR Language Model for Decoding Untranslated Regions of mRNA and Function Predictions](https://mlsanity.com/paper/2310.03281/) (2023)
- [BERTs are Generative In-Context Learners](https://mlsanity.com/paper/2406.04823/) (2024)
- [The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources](https://mlsanity.com/paper/2406.16746/) (2024)
- [Technical Note on Transcription Factor Motif Discovery from Importance Scores (TF-MoDISco) version 0.5.6.5](https://mlsanity.com/paper/1811.00416/) (2018)

## Built on by (most cited first, 20 of 20)

- [BioReason: Incentivizing Multimodal Biological Reasoning within a DNA-LLM Model](https://mlsanity.com/paper/2505.23579/) (2025)
- [Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey](https://mlsanity.com/paper/2412.18619/) (2024)
- [JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures](https://mlsanity.com/paper/2602.17162/) (2026)
- [EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants](https://mlsanity.com/paper/2502.20309/) (2025)
- [JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model](https://mlsanity.com/paper/2505.17257/) (2025)
- [Open-weight genome language model safeguards: Assessing robustness via adversarial fine-tuning](https://mlsanity.com/paper/2511.19299/) (2025)
- [Diverse capability and scaling of diffusion and auto-regressive models when learning abstract rules](https://mlsanity.com/paper/2411.07873/) (2024)
- [A Phylogenetic Approach to Genomic Language Modeling](https://mlsanity.com/paper/2503.03773/) (2025)
- [Gene42: Long-Range Genomic Foundation Model With Dense Attention](https://mlsanity.com/paper/2503.16565/) (2025)
- [Foundation Models for AI-Enabled Biological Design](https://mlsanity.com/paper/2505.11610/) (2025)
- [Biology-informed neural networks learn nonlinear representations from omics data to improve genomic prediction and interpretability](https://mlsanity.com/paper/2510.14970/) (2025)
- [NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction](https://mlsanity.com/paper/2511.02888/) (2025)
- [Genomic Next-Token Predictors are In-Context Learners](https://mlsanity.com/paper/2511.12797/) (2025)
- [Can linguists better understand DNA?](https://mlsanity.com/paper/2412.07678/) (2024)
- [CodonMoE: DNA Language Models for mRNA Analyses](https://mlsanity.com/paper/2508.04739/) (2025)
- [A Deep Learning Pipeline for Epilepsy Genomic Analysis Using GPT-2 XL and NVIDIA H100](https://mlsanity.com/paper/2510.00392/) (2025)
- [Artificial Intelligence Virtual Cells: From Measurements to Decisions across Modality, Scale, Dynamics, and Evaluation](https://mlsanity.com/paper/2510.12498/) (2025)
- [AntigenLM: Structure-Aware DNA Language Modeling for Influenza](https://mlsanity.com/paper/2602.09067/) (2026)
- [GenomeOcean Anywhere: Private WebGPU Inference for Genome MoEs](https://mlsanity.com/paper/2609.35882/) (2026)
- [VANDAM: Viewing a nucleotide sequence with DNA molecular priors](https://mlsanity.com/paper/2610.00411/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are having transformative impacts across a wide range of scientific fields, particularly in the biomedical sciences. Just as the goal of Natural Language Processing is to understand sequences of words, a major objective in biology is to understand biological sequences. Genomic Language Models (gLMs), which are LLMs trained on DNA sequences, have the potential to significantly advance our understanding of genomes and how DNA elements at various scales interact to give rise to complex functions. To showcase this potential, we highlight key applications of gLMs, including functional constraint prediction, sequence design, and transfer learning. Despite notable recent progress, however, developing effective and efficient gLMs presents numerous challenges, especially for species with large, complex genomes. Here, we discuss major considerations for developing and evaluating gLMs.