mlsanityData to 7 Oct 2026

SNP2Vec: Scalable Self-Supervised Pre-Training for Genome-Wide Association Study

arXiv 2204.06699 ↗First posted 14 Apr 2022cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Self-supervised pre-training methods have brought remarkable breakthroughs in the understanding of text, image, and speech. Recent developments in genomics has also adopted these pre-training methods for genome understanding. However, they focus only on understanding haploid sequences, which hinders their applicability towards understanding genetic variations, also known as single nucleotide polymorphisms (SNPs), which is crucial for genome-wide association study. In this paper, we introduce SNP2Vec, a scalable self-supervised pre-training approach for understanding SNP. We apply SNP2Vec to perform long-sequence genomics modeling, and we evaluate the effectiveness of our approach on predicting Alzheimer's disease risk in a Chinese cohort. Our approach significantly outperforms existing polygenic risk score methods and all other baselines, including the model that is trained entirely with haploid sequences. We release our code and dataset on https://github.com/HLTCHKUST/snp2vec.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 18 of its 57 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

18 of 57 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
6 more
  1. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  2. 2020ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computingcited by 162
  3. 2020ProGen: Language Modeling for Protein Generationcited by 127
  4. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  5. 2021Are Multilingual Models Effective in Code-Switching?cited by 56
  6. 2021AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarizationcited by 51

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 3.

5 papers.

  1. 2024High-Dimension Human Value Representation in Large Language Modelscited by 20
  2. 2024CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scalecited by 18
  3. 2025BMFM-DNA: A SNP-aware DNA foundation model to capture variant effectscited by 2
  4. 2025Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysiscited by 0
  5. 2025HEMERA: A Human-Explainable Transformer Model for Estimating Lung Cancer Risk using GWAS Datacited by 0