mlsanityData to 14 Oct 2026

Language modelling techniques for analysing the impact of human genetic variation

arXiv 2503.10655 ↗First posted 7 Mar 2025cs.CL cs.AI q-bio.BMcited by 0

Not on the map. Why?

Abstract (arXiv)

Interpreting the effects of variants within the human genome and proteome is essential for analysing disease risk, predicting medication response, and developing personalised health interventions. Due to the intrinsic similarities between the structure of natural languages and genetic sequences, natural language processing techniques have demonstrated great applicability in computational variant effect prediction. In particular, the advent of the Transformer has led to significant advancements in the field. However, Transformer-based models are not without their limitations, and a number of extensions and alternatives have been developed to improve results and enhance computational efficiency. This review explores the use of language models for computational variant effect prediction over the past decade, analysing the main architectures, and identifying key trends and future directions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 50 of its 283 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

50 of 283 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
38 more
  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  5. 2019Attention is not Explanationcited by 929
  6. 2019Attention is not not Explanationcited by 636
  7. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  8. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  9. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  10. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  11. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 194
  12. 2020The Computational Limits of Deep Learningcited by 193
  13. 2022Talking About Large Language Modelscited by 175
  14. 2023DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomecited by 143
  15. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  16. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 130
  17. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 125
  18. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  19. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 113
  20. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  21. 2017Structured Attention Networkscited by 95
  22. 2022Tranception: protein fitness prediction with autoregressive transformers and inference-time retrievalcited by 79
  23. 2021Language Model Evaluation Beyond Perplexitycited by 78
  24. 2021Membership Inference Attack Susceptibility of Clinical Language Modelscited by 52
  25. 2018Predicting the Computational Cost of Deep Learning Modelscited by 49
  26. 2019ProteinNet: a standardized data set for machine learning of protein structurecited by 39
  27. 2023User Inference Attacks on Large Language Modelscited by 36
  28. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  29. 2023PoET: A generative model of protein families as sequences-of-sequencescited by 29
  30. 2024Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysiscited by 21
  31. 2023Zero-shot causal learningcited by 17
  32. 2023To Transformers and Beyond: Large Language Models for the Genomecited by 15
  33. 2024Progress and Opportunities of Foundation Models in Bioinformaticscited by 13
  34. 2024Efficient and Personalized Mobile Health Event Prediction via Small Language Modelscited by 6
  35. 2024Fine-tuning Protein Language Models with Deep Mutational Scanning improves Variant Effect Predictioncited by 3
  36. 2021Deciphering the Language of Nature: A transformer-based language model for deleterious mutations in proteinscited by 2
  37. 2024DYNA: Disease-Specific Language Model for Variant Pathogenicitycited by 2
  38. 2022Unsupervised language models for disease variant predictioncited by 1

Built on by

Not cited yet.