# ReLSO: A Transformer-based Model for Latent Space Optimization and Generation of Proteins

arXiv: https://arxiv.org/abs/2201.09948 · HTML: https://mlsanity.com/paper/2201.09948/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 33 references that are in the map.
- First posted on arXiv 24 Jan 2022.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [Spectral Norm Regularization for Improving the Generalizability of Deep Learning](https://mlsanity.com/paper/1705.10941/) (2017)
- [Conditioning by adaptive sampling for robust design](https://mlsanity.com/paper/1901.10060/) (2019)
- [BERTology Meets Biology: Interpreting Attention in Protein Language Models](https://mlsanity.com/paper/2006.15222/) (2020)
- [Design by adaptive sampling](https://mlsanity.com/paper/1810.03714/) (2018)
- [Fast differentiable DNA and protein sequence optimization for molecular design](https://mlsanity.com/paper/2005.11275/) (2020)

## Built on by (most cited first, 4 of 4)

- [Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMC](https://mlsanity.com/paper/2212.09925/) (2022)
- [Robust Optimization in Protein Fitness Landscapes Using Reinforcement Learning in Latent Space](https://mlsanity.com/paper/2405.18986/) (2024)
- [Generative AI for Controllable Protein Sequence Design: A Survey](https://mlsanity.com/paper/2402.10516/) (2024)
- [Protein Counterfactuals via Diffusion-Guided Latent Optimization](https://mlsanity.com/paper/2603.10811/) (2026)

## Abstract (arXiv)

The development of powerful natural language models have increased the ability to learn meaningful representations of protein sequences. In addition, advances in high-throughput mutagenesis, directed evolution, and next-generation sequencing have allowed for the accumulation of large amounts of labeled fitness data. Leveraging these two trends, we introduce Regularized Latent Space Optimization (ReLSO), a deep transformer-based autoencoder which features a highly structured latent space that is trained to jointly generate sequences as well as predict fitness. Through regularized prediction heads, ReLSO introduces a powerful protein sequence encoder and novel approach for efficient fitness landscape traversal. Using ReLSO, we explicitly model the sequence-function landscape of large labeled datasets and generate new molecules by optimizing within the latent space using gradient-based methods. We evaluate this approach on several publicly-available protein datasets, including variant sets of anti-ranibizumab and GFP. We observe a greater sequence optimization efficiency (increase in fitness per optimization step) by ReLSO compared to other approaches, where ReLSO more robustly generates high-fitness sequences. Furthermore, the attention-based relationships learned by the jointly-trained ReLSO models provides a potential avenue towards sequence-level fitness attribution information.