# Structure Language Models for Protein Conformation Generation

arXiv: https://arxiv.org/abs/2410.18403 · HTML: https://mlsanity.com/paper/2410.18403/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 76 references that are in the map.
- First posted on arXiv 24 Oct 2024.

## Builds on (most cited first, 29 of 29)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [Learning from Protein Structure with Geometric Vector Perceptrons](https://mlsanity.com/paper/2009.01411/) (2020)
- [Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densities](https://mlsanity.com/paper/2006.02425/) (2020)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [Score-based Continuous-time Discrete Diffusion Models](https://mlsanity.com/paper/2211.16750/) (2022)
- [A Reparameterized Discrete Diffusion Model for Text Generation](https://mlsanity.com/paper/2302.05737/) (2023)
- [AlphaFold Meets Flow Matching for Generating Protein Ensembles](https://mlsanity.com/paper/2402.04845/) (2024)
- [PiFold: Toward effective and efficient protein inverse folding](https://mlsanity.com/paper/2209.12643/) (2022)
- [Two for One: Diffusion Models and Force Fields for Coarse-Grained Molecular Dynamics](https://mlsanity.com/paper/2302.00600/) (2023)
- [Timewarp: Transferable Acceleration of Molecular Dynamics by Learning Time-Coarsened Dynamics](https://mlsanity.com/paper/2302.01170/) (2023)
- [EigenFold: Generative Protein Structure Prediction with Diffusion Models](https://mlsanity.com/paper/2304.02198/) (2023)
- [Generative Modeling of Molecular Dynamics Trajectories](https://mlsanity.com/paper/2409.17808/) (2024)
- [Protein Conformation Generation via Force-Guided SE(3) Diffusion Models](https://mlsanity.com/paper/2403.14088/) (2024)
- [Deeptime: a Python library for machine learning dynamical models from time series data](https://mlsanity.com/paper/2110.15013/) (2021)
- [Str2Str: A Score-based Framework for Zero-shot Protein Conformation Sampling](https://mlsanity.com/paper/2306.03117/) (2023)
- [Towards Predicting Equilibrium Distributions for Molecular Systems with Deep Learning](https://mlsanity.com/paper/2306.05445/) (2023)
- [FoldToken: Learning Protein Language via Vector Quantization and Beyond](https://mlsanity.com/paper/2403.09673/) (2024)
- [Fusing Neural and Physical: Augment Protein Conformation Sampling with Tractable Simulations](https://mlsanity.com/paper/2402.10433/) (2024)

## Built on by (most cited first, 17 of 17)

- [SimpleFold: Folding Proteins is Simpler than You Think](https://mlsanity.com/paper/2509.18480/) (2025)
- [ConfRover: Simultaneous Modeling of Protein Conformation and Dynamics via Autoregression](https://mlsanity.com/paper/2505.17478/) (2025)
- [MarS-FM: Generative Modeling of Molecular Dynamics via Markov State Models](https://mlsanity.com/paper/2509.24779/) (2025)
- [Aligning Protein Conformation Ensemble Generation with Physical Feedback](https://mlsanity.com/paper/2505.24203/) (2025)
- [Learning residue level protein dynamics with multiscale Gaussians](https://mlsanity.com/paper/2509.01038/) (2025)
- [Reaction-conditioned De Novo Enzyme Design with GENzyme](https://mlsanity.com/paper/2411.16694/) (2024)
- [TEMPO: Temporal Multi-scale Autoregressive Generation of Protein Conformational Ensembles](https://mlsanity.com/paper/2511.05510/) (2025)
- [Atomic Trajectory Modeling with State Space Models for Biomolecular Dynamics](https://mlsanity.com/paper/2603.17633/) (2026)
- [Learning conformational ensembles of proteins based on backbone geometry](https://mlsanity.com/paper/2503.05738/) (2025)
- [AI-based Methods for Simulating, Sampling, and Predicting Protein Ensembles](https://mlsanity.com/paper/2509.17224/) (2025)
- [Learning Structure, Energy, and Dynamics: A Survey of Artificial Intelligence for Protein Dynamics](https://mlsanity.com/paper/2604.25244/) (2026)
- [ENSEMBITS: an alphabet of protein conformational ensembles](https://mlsanity.com/paper/2605.13789/) (2026)
- [SimpleDesign: A Joint Model for Protein Sequence and Structure Codesign](https://mlsanity.com/paper/2609.03377/) (2026)
- [RiboSphere: Learning Unified and Efficient Representations of RNA Structures](https://mlsanity.com/paper/2603.19636/) (2026)
- [ConforNets: Latents-Based Conformational Control in OpenFold3](https://mlsanity.com/paper/2604.18559/) (2026)
- [Co-Generative De Novo Functional Protein Design](https://mlsanity.com/paper/2605.00948/) (2026)
- [Learning Latent Protein Languages for Autoregressive Generation](https://mlsanity.com/paper/2610.03978/) (2026)

## Abstract (arXiv)

Proteins adopt multiple structural conformations to perform their diverse biological functions, and understanding these conformations is crucial for advancing drug discovery. Traditional physics-based simulation methods often struggle with sampling equilibrium conformations and are computationally expensive. Recently, deep generative models have shown promise in generating protein conformations as a more efficient alternative. However, these methods predominantly rely on the diffusion process within a 3D geometric space, which typically centers around the vicinity of metastable states and is often inefficient in terms of runtime. In this paper, we introduce Structure Language Modeling (SLM) as a novel framework for efficient protein conformation generation. Specifically, the protein structures are first encoded into a compact latent space using a discrete variational auto-encoder, followed by conditional language modeling that effectively captures sequence-specific conformation distributions. This enables a more efficient and interpretable exploration of diverse ensemble modes compared to existing methods. Based on this general framework, we instantiate SLM with various popular LM architectures as well as proposing the ESMDiff, a novel BERT-like structure language model fine-tuned from ESM3 with masked diffusion. We verify our approach in various scenarios, including the equilibrium dynamics of BPTI, conformational change pairs, and intrinsically disordered proteins. SLM provides a highly efficient solution, offering a 20-100x speedup than existing methods in generating diverse conformations, shedding light on promising avenues for future research.