# Generative power of a protein language model trained on multiple sequence alignments

arXiv: https://arxiv.org/abs/2204.07110 · HTML: https://mlsanity.com/paper/2204.07110/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 94 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [ProGen: Language Modeling for Protein Generation](https://mlsanity.com/paper/2004.03497/) (2020)
- [BERTology Meets Biology: Interpreting Attention in Protein Language Models](https://mlsanity.com/paper/2006.15222/) (2020)
- [Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings](https://mlsanity.com/paper/2106.02736/) (2021)
- [Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information](https://mlsanity.com/paper/1902.11239/) (2019)
- [Protein language models trained on multiple sequence alignments learn phylogenetic relationships](https://mlsanity.com/paper/2203.15465/) (2022)
- [Generative Capacity of Probabilistic Protein Sequence Models](https://mlsanity.com/paper/2012.02296/) (2020)

## Built on by (most cited first, 9 of 9)

- [A Text-guided Protein Design Framework](https://mlsanity.com/paper/2302.04611/) (2023)
- [OpenProteinSet: Training data for structural biology at scale](https://mlsanity.com/paper/2308.05326/) (2023)
- [Generative AI for Controllable Protein Sequence Design: A Survey](https://mlsanity.com/paper/2402.10516/) (2024)
- [Computational Protein Science in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2501.10282/) (2025)
- [Enhancing the Protein Tertiary Structure Prediction by Multiple Sequence Alignment Generation](https://mlsanity.com/paper/2306.01824/) (2023)
- [Large Language Models for Bioinformatics](https://mlsanity.com/paper/2501.06271/) (2025)
- [Accurate Estimation of Mutual Information in High Dimensional Data](https://mlsanity.com/paper/2506.00330/) (2025)
- [Pairing interacting protein sequences using masked language modeling](https://mlsanity.com/paper/2308.07136/) (2023)
- [Computational design of target-specific linear peptide binders with TransformerBeta](https://mlsanity.com/paper/2410.16302/) (2024)

## Abstract (arXiv)

Computational models starting from large ensembles of evolutionarily related protein sequences capture a representation of protein families and learn constraints associated to protein structure and function. They thus open the possibility for generating novel sequences belonging to protein families. Protein language models trained on multiple sequence alignments, such as MSA Transformer, are highly attractive candidates to this end. We propose and test an iterative method that directly employs the masked language modeling objective to generate sequences using MSA Transformer. We demonstrate that the resulting sequences score as well as natural sequences, for homology, coevolution and structure-based measures. For large protein families, our synthetic sequences have similar or better properties compared to sequences generated by Potts models, including experimentally-validated ones. Moreover, for small protein families, our generation method based on MSA Transformer outperforms Potts models. Our method also more accurately reproduces the higher-order statistics and the distribution of sequences in sequence space of natural data than Potts models. MSA Transformer is thus a strong candidate for protein sequence generation and protein design.