# Retrieved Sequence Augmentation for Protein Representation Learning

arXiv: https://arxiv.org/abs/2302.12563 · HTML: https://mlsanity.com/paper/2302.12563/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 89 references that are in the map.
- First posted on arXiv 24 Feb 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [ProGen2: Exploring the Boundaries of Protein Language Models](https://mlsanity.com/paper/2206.13517/) (2022)
- [Efficient Nearest Neighbor Language Models](https://mlsanity.com/paper/2109.04212/) (2021)
- [Adaptive Semiparametric Language Models](https://mlsanity.com/paper/2102.02557/) (2021)
- [PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding](https://mlsanity.com/paper/2206.02096/) (2022)
- [OntoProtein: Protein Pretraining With Gene Ontology Embedding](https://mlsanity.com/paper/2201.11147/) (2022)
- [Retrieval-Augmented Reinforcement Learning](https://mlsanity.com/paper/2202.08417/) (2022)
- [DeepSF: deep convolutional neural network for mapping protein sequences to folds](https://mlsanity.com/paper/1706.01010/) (2017)
- [ProteinNet: a standardized data set for machine learning of protein structure](https://mlsanity.com/paper/1902.00249/) (2019)
- [Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model](https://mlsanity.com/paper/2110.15527/) (2021)
- [Exploring evolution-aware & -free protein language models as protein function predictors](https://mlsanity.com/paper/2206.06583/) (2022)
- [Deep Recurrent Neural Network for Protein Function Prediction from Sequence](https://mlsanity.com/paper/1701.08318/) (2017)
- [Generalization Properties of Retrieval-based Models](https://mlsanity.com/paper/2210.02617/) (2022)
- [Aligning 415 519 proteins in less than two hours on PC](https://mlsanity.com/paper/1603.06958/) (2016)

## Built on by (most cited first, 7 of 7)

- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [A Survey of Generative AI for de novo Drug Design: New Frontiers in Molecule and Protein Generation](https://mlsanity.com/paper/2402.08703/) (2024)
- [Protriever: End-to-End Differentiable Protein Homology Search for Fitness Prediction](https://mlsanity.com/paper/2506.08954/) (2025)
- [GOProteinGNN: Leveraging Protein Knowledge Graphs for Protein Representation Learning](https://mlsanity.com/paper/2408.00057/) (2024)
- [Knowledge-Augmented Long-CoT Generation for Complex Biomolecular Reasoning](https://mlsanity.com/paper/2511.08024/) (2025)
- [ProtIR: Iterative Refinement between Retrievers and Predictors for Protein Function Annotation](https://mlsanity.com/paper/2402.07955/) (2024)
- [Unlocking Biological Workflows for Robust Protein-Text Question Answering: A Dual-Dimensional RAG Framework](https://mlsanity.com/paper/2605.17261/) (2026)

## Abstract (arXiv)

Protein language models have excelled in a variety of tasks, ranging from structure prediction to protein engineering. However, proteins are highly diverse in functions and structures, and current state-of-the-art models including the latest version of AlphaFold rely on Multiple Sequence Alignments (MSA) to feed in the evolutionary knowledge. Despite their success, heavy computational overheads, as well as the de novo and orphan proteins remain great challenges in protein representation learning. In this work, we show that MSAaugmented models inherently belong to retrievalaugmented methods. Motivated by this finding, we introduce Retrieved Sequence Augmentation(RSA) for protein representation learning without additional alignment or pre-processing. RSA links query protein sequences to a set of sequences with similar structures or properties in the database and combines these sequences for downstream prediction. We show that protein language models benefit from the retrieval enhancement on both structure prediction and property prediction tasks, with a 5% improvement on MSA Transformer on average while being 373 times faster. In addition, we show that our model can transfer to new protein domains better and outperforms MSA Transformer on de novo protein prediction. Our study fills a much-encountered gap in protein prediction and brings us a step closer to demystifying the domain knowledge needed to understand protein sequences. Code is available on https://github.com/HKUNLP/RSA.