# Protein sequence design with deep generative models

arXiv: https://arxiv.org/abs/2104.04457 · HTML: https://mlsanity.com/paper/2104.04457/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 87 references that are in the map.
- First posted on arXiv 9 Apr 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [Adversarially Learned Inference](https://mlsanity.com/paper/1606.00704/) (2016)
- [ProGen: Language Modeling for Protein Generation](https://mlsanity.com/paper/2004.03497/) (2020)
- [Conditioning by adaptive sampling for robust design](https://mlsanity.com/paper/1901.10060/) (2019)
- [Learning protein sequence embeddings using information from structure](https://mlsanity.com/paper/1902.08661/) (2019)
- [Autofocused oracles for model-based design](https://mlsanity.com/paper/2006.08052/) (2020)
- [A Hybrid Convolutional Variational Autoencoder for Text Generation](https://mlsanity.com/paper/1702.02390/) (2017)
- [Design by adaptive sampling](https://mlsanity.com/paper/1810.03714/) (2018)
- [Is Transfer Learning Necessary for Protein Landscape Prediction?](https://mlsanity.com/paper/2011.03443/) (2020)

## Built on by (most cited first, 30 of 35)

- [PiFold: Toward effective and efficient protein inverse folding](https://mlsanity.com/paper/2209.12643/) (2022)
- [A Text-guided Protein Design Framework](https://mlsanity.com/paper/2302.04611/) (2023)
- [Learning Hierarchical Protein Representations via Complete 3D Graph Networks](https://mlsanity.com/paper/2207.12600/) (2022)
- [Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design](https://mlsanity.com/paper/2202.03613/) (2022)
- [Regression Transformer: Concurrent sequence regression and generation for molecular language modeling](https://mlsanity.com/paper/2202.01338/) (2022)
- [A Latent Diffusion Model for Protein Structure Generation](https://mlsanity.com/paper/2305.04120/) (2023)
- [Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples](https://mlsanity.com/paper/2302.04440/) (2023)
- [Deep Extrapolation for Attribute-Enhanced Generation](https://mlsanity.com/paper/2107.02968/) (2021)
- [Deep Generative Modeling for Protein Design](https://mlsanity.com/paper/2109.13754/) (2021)
- [Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMC](https://mlsanity.com/paper/2212.09925/) (2022)
- [De novo protein design using geometric vector field networks](https://mlsanity.com/paper/2310.11802/) (2023)
- [Adaptive machine learning for protein engineering](https://mlsanity.com/paper/2106.05466/) (2021)
- [Knowledge-Design: Pushing the Limit of Protein Design via Knowledge Refinement](https://mlsanity.com/paper/2305.15151/) (2023)
- [Diffusion on language model encodings for protein sequence generation](https://mlsanity.com/paper/2403.03726/) (2024)
- [DiffSDS: A language diffusion model for protein backbone inpainting under geometric conditions and constraints](https://mlsanity.com/paper/2301.09642/) (2023)
- [A Multi-Modal Contrastive Diffusion Model for Therapeutic Peptide Generation](https://mlsanity.com/paper/2312.15665/) (2023)
- [Generative AI for Controllable Protein Sequence Design: A Survey](https://mlsanity.com/paper/2402.10516/) (2024)
- [Thompson Sampling with Diffusion Generative Prior](https://mlsanity.com/paper/2301.05182/) (2023)
- [Steering Generative Models with Experimental Data for Protein Fitness Optimization](https://mlsanity.com/paper/2505.15093/) (2025)
- [Diffusion map particle systems for generative modeling](https://mlsanity.com/paper/2304.00200/) (2023)
- [De novo antibody design with SE(3) diffusion](https://mlsanity.com/paper/2405.07622/) (2024)
- [Mask prior-guided denoising diffusion improves inverse protein folding](https://mlsanity.com/paper/2412.07815/) (2024)
- [A theoretical framework for overfitting in energy-based modeling](https://mlsanity.com/paper/2501.19158/) (2025)
- [α-TCVAE: On the relationship between Disentanglement and Diversity](https://mlsanity.com/paper/2411.00588/) (2024)
- [Function-Guided Conditional Generation Using Protein Language Models with Adapters](https://mlsanity.com/paper/2410.03634/) (2024)
- [De Novo Drug Design with Joint Transformers](https://mlsanity.com/paper/2310.02066/) (2023)
- [Discrete Compositional Generation via General Soft Operators and Robust Reinforcement Learning](https://mlsanity.com/paper/2506.17007/) (2025)
- [Target specific peptide design using latent space approximate trajectory collector](https://mlsanity.com/paper/2302.01435/) (2023)
- [ProtFIM: Fill-in-Middle Protein Sequence Design via Protein Language Models](https://mlsanity.com/paper/2303.16452/) (2023)
- [Graph Representation Learning for Interactive Biomolecule Systems](https://mlsanity.com/paper/2304.02656/) (2023)

## Abstract (arXiv)

Protein engineering seeks to identify protein sequences with optimized properties. When guided by machine learning, protein sequence generation methods can draw on prior knowledge and experimental efforts to improve this process. In this review, we highlight recent applications of machine learning to generate protein sequences, focusing on the emerging field of deep generative methods.