# Unsupervised language models for disease variant prediction

arXiv: https://arxiv.org/abs/2212.03979 · HTML: https://mlsanity.com/paper/2212.03979/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 28 references that are in the map.
- First posted on arXiv 7 Dec 2022.

## Builds on (most cited first, 3 of 3)

- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [ProGen: Language Modeling for Protein Generation](https://mlsanity.com/paper/2004.03497/) (2020)
- [Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval](https://mlsanity.com/paper/2205.13760/) (2022)

## Built on by (most cited first, 1 of 1)

- [Language modelling techniques for analysing the impact of human genetic variation](https://mlsanity.com/paper/2503.10655/) (2025)

## Abstract (arXiv)

There is considerable interest in predicting the pathogenicity of protein variants in human genes. Due to the sparsity of high quality labels, recent approaches turn to textitunsupervised learning, using Multiple Sequence Alignments (MSAs) to train generative models of natural sequence variation within each gene. These generative models then predict variant likelihood as a proxy to evolutionary fitness. In this work we instead combine this evolutionary principle with pretrained protein language models (LMs), which have already shown promising results in predicting protein structure and function. Instead of training separate models per-gene, we find that a single protein LM trained on broad sequence datasets can score pathogenicity for any gene variant zero-shot, without MSAs or finetuning. We call this unsupervised approach VELM (Variant Effect via Language Models), and show that it achieves scoring performance comparable to the state of the art when evaluated on clinically labeled variants of disease-related genes.