# MSAGPT: Neural Prompting Protein Structure Prediction via MSA Generative Pre-Training

arXiv: https://arxiv.org/abs/2406.05347 · HTML: https://mlsanity.com/paper/2406.05347/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 52 references that are in the map.
- First posted on arXiv 8 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [ProGen2: Exploring the Boundaries of Protein Language Models](https://mlsanity.com/paper/2206.13517/) (2022)
- [xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein](https://mlsanity.com/paper/2401.06199/) (2024)
- [PoET: A generative model of protein families as sequences-of-sequences](https://mlsanity.com/paper/2306.06156/) (2023)
- [Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models](https://mlsanity.com/paper/2012.00195/) (2020)
- [OpenProteinSet: Training data for structural biology at scale](https://mlsanity.com/paper/2308.05326/) (2023)
- [Enhancing the Protein Tertiary Structure Prediction by Multiple Sequence Alignment Generation](https://mlsanity.com/paper/2306.01824/) (2023)
- [Unsupervisedly Prompting AlphaFold2 for Few-Shot Learning of Accurate Folding Landscape and Protein Structure Prediction](https://mlsanity.com/paper/2208.09652/) (2022)

## Built on by (most cited first, 7 of 7)

- [Computational Protein Science in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2501.10282/) (2025)
- [Advances in Protein Representation Learning: Methods, Applications, and Future Directions](https://mlsanity.com/paper/2503.16659/) (2025)
- [ProteinAE: Protein Diffusion Autoencoders for Structure Encoding](https://mlsanity.com/paper/2510.10634/) (2025)
- [Lightweight MSA Design Advances Protein Folding From Evolutionary Embeddings](https://mlsanity.com/paper/2507.07032/) (2025)
- [MID: A Self-supervised Multimodal Iterative Denoising Framework](https://mlsanity.com/paper/2511.00997/) (2025)
- [Rigidity-Aware Geometric Pretraining for Protein Design and Conformational Ensembles](https://mlsanity.com/paper/2603.02406/) (2026)
- [From Snapshots to Symphonies: The Evolution of Protein Prediction from Static Structures to Generative Dynamics and Multimodal Interactions](https://mlsanity.com/paper/2603.18505/) (2026)

## Abstract (arXiv)

Multiple Sequence Alignment (MSA) plays a pivotal role in unveiling the evolutionary trajectories of protein families. The accuracy of protein structure predictions is often compromised for protein sequences that lack sufficient homologous information to construct high quality MSA. Although various methods have been proposed to generate virtual MSA under these conditions, they fall short in comprehensively capturing the intricate coevolutionary patterns within MSA or require guidance from external oracle models. Here we introduce MSAGPT, a novel approach to prompt protein structure predictions via MSA generative pretraining in the low MSA regime. MSAGPT employs a simple yet effective 2D evolutionary positional encoding scheme to model complex evolutionary patterns. Endowed by this, its flexible 1D MSA decoding framework facilitates zero or few shot learning. Moreover, we demonstrate that leveraging the feedback from AlphaFold2 can further enhance the model capacity via Rejective Fine tuning (RFT) and Reinforcement Learning from AF2 Feedback (RLAF). Extensive experiments confirm the efficacy of MSAGPT in generating faithful virtual MSA to enhance the structure prediction accuracy. The transfer learning capabilities also highlight its great potential for facilitating other protein tasks.