mlsanityData to 7 Oct 2026

Genomic Language Models: Opportunities and Challenges

arXiv 2407.11435 ↗First posted 16 Jul 2024q-bio.GN cs.LG stat.MLcited by 20

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are having transformative impacts across a wide range of scientific fields, particularly in the biomedical sciences. Just as the goal of Natural Language Processing is to understand sequences of words, a major objective in biology is to understand biological sequences. Genomic Language Models (gLMs), which are LLMs trained on DNA sequences, have the potential to significantly advance our understanding of genomes and how DNA elements at various scales interact to give rise to complex functions. To showcase this potential, we highlight key applications of gLMs, including functional constraint prediction, sequence design, and transfer learning. Despite notable recent progress, however, developing effective and efficient gLMs presents numerous challenges, especially for species with large, complex genomes. Here, we discuss major considerations for developing and evaluating gLMs.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 32 of its 135 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

32 of 135 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
20 more
  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  4. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  5. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  6. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  7. 2020A Survey on Neural Network Interpretabilitycited by 291
  8. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 187
  9. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  10. 2023DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomecited by 139
  11. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
  12. 2023PoET: A generative model of protein families as sequences-of-sequencescited by 29
  13. 2021Are Pre-trained Convolutions Better than Pre-trained Transformers?cited by 27
  14. 2023BEND: Benchmarking DNA Language Models on biologically meaningful taskscited by 27
  15. 2023A 5' UTR Language Model for Decoding Untranslated Regions of mRNA and Function Predictionscited by 20
  16. 2024BERTs are Generative In-Context Learnerscited by 11
  17. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  18. 2018Technical Note on Transcription Factor Motif Discovery from Importance Scores (TF-MoDISco) version 0.5.6.5cited by 7
  19. 2024Multi-modal Transfer Learning between Biological Foundation Modelscited by 6
  20. 2021Epigenomic language models powered by Cerebrascited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 4.

20 papers.

Sort:

By area:

  1. 2025BioReason: Incentivizing Multimodal Biological Reasoning within a DNA-LLM Modelcited by 35
  2. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  3. 2026JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecturescited by 9
  4. 2025EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistantscited by 5
  5. 2025JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Modelcited by 4
  6. 2025Open-weight genome language model safeguards: Assessing robustness via adversarial fine-tuningcited by 4
  7. 2024Diverse capability and scaling of diffusion and auto-regressive models when learning abstract rulescited by 3
  8. 2025A Phylogenetic Approach to Genomic Language Modelingcited by 2
  9. 2025Gene42: Long-Range Genomic Foundation Model With Dense Attentioncited by 1
  10. 2025Foundation Models for AI-Enabled Biological Designcited by 1
  11. 2025Biology-informed neural networks learn nonlinear representations from omics data to improve genomic prediction and interpretabilitycited by 1
  12. 2025NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Predictioncited by 1
8 more
  1. 2025Genomic Next-Token Predictors are In-Context Learnerscited by 1
  2. 2024Can linguists better understand DNA?cited by 0
  3. 2025CodonMoE: DNA Language Models for mRNA Analysescited by 0
  4. 2025A Deep Learning Pipeline for Epilepsy Genomic Analysis Using GPT-2 XL and NVIDIA H100cited by 0
  5. 2025Artificial Intelligence Virtual Cells: From Measurements to Decisions across Modality, Scale, Dynamics, and Evaluationcited by 0
  6. 2026AntigenLM: Structure-Aware DNA Language Modeling for Influenzacited by 0
  7. 2026GenomeOcean Anywhere: Private WebGPU Inference for Genome MoEscited by 0
  8. 2026VANDAM: Viewing a nucleotide sequence with DNA molecular priorscited by 0