mlsanityData to 7 Oct 2026

To Transformers and Beyond: Large Language Models for the Genome

arXiv 2311.07621 ↗First posted 13 Nov 2023q-bio.GN cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

In the rapidly evolving landscape of genomics, deep learning has emerged as a useful tool for tackling complex computational challenges. This review focuses on the transformative role of Large Language Models (LLMs), which are mostly based on the transformer architecture, in genomics. Building on the foundation of traditional convolutional neural networks and recurrent neural networks, we explore both the strengths and limitations of transformers and other LLMs for genomics. Additionally, we contemplate the future of genomic modeling beyond the transformer architecture based on current trends in research. The paper aims to serve as a guide for computational biologists and computer scientists interested in LLMs for genomic data. We hope the paper can also serve as an educational introduction and discussion for biologists to a fundamental shift in how we will be analyzing genomic data in the future.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 38 of its 129 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

38 of 129 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
26 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2018Group Normalizationcited by 1,514
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2021Transformers in Vision: A Surveycited by 1,009
  8. 2022Token Merging: Your ViT But Fastercited by 995
  9. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  10. 2020Quantifying Attention Flow in Transformerscited by 759
  11. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  12. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  13. 2021A Survey of Transformerscited by 399
  14. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  15. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 197
  16. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 187
  17. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  18. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  19. 2023DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomecited by 139
  20. 2022Dilated Neighborhood Attention Transformercited by 64
  21. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  22. 2014A survey of dimensionality reduction techniquescited by 50
  23. 2022WeightedSHAP: analyzing and improving Shapley based feature attributionscited by 36
  24. 2021Multi-head or Single-head? An Empirical Comparison for Transformer Trainingcited by 15
  25. 2016DeeperBind: Enhancing Prediction of Sequence Specificities of DNA Binding Proteinscited by 7
  26. 2022Application of Deep Learning on Single-Cell RNA-sequencing Data Analysis: A Reviewcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 8; 2025: 7.

15 papers.

Sort:

By area:

  1. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  2. 2024Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trialscited by 24
  3. 2024Combining Knowledge Graphs and Large Language Modelscited by 16
  4. 2024Large Language Models in Plant Biologycited by 8
  5. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
  6. 2025Computational Protein Science in the Era of Large Language Models (LLMs)cited by 6
  7. 2024BarcodeMamba: State Space Models for Biodiversity Analysiscited by 4
  8. 2025JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Modelcited by 4
  9. 2025Hyperbolic Genome Embeddingscited by 3
  10. 2024Unlocking Efficiency: Adaptive Masking for Gene Transformer Modelscited by 2
  11. 2024Whole Genome Transformer for Gene Interaction Effects in Microbiome Habitat Specificitycited by 1
  12. 2024Assessing Foundation Models' Transferability to Physiological Signals in Precision Medicinecited by 1
3 more
  1. 2025Gene42: Long-Range Genomic Foundation Model With Dense Attentioncited by 1
  2. 2025Language modelling techniques for analysing the impact of human genetic variationcited by 0
  3. 2025Fractal Language Modelling by Universal Sequence Maps (USM)cited by 0