# DPLM-2: A Multimodal Diffusion Protein Language Model

arXiv: https://arxiv.org/abs/2410.13782 · HTML: https://mlsanity.com/paper/2410.13782/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 85 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021)
- [Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions](https://mlsanity.com/paper/2209.11215/) (2022)
- [Learning from Protein Structure with Geometric Vector Perceptrons](https://mlsanity.com/paper/2009.01411/) (2020)
- [Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design](https://mlsanity.com/paper/2402.04997/) (2024)
- [Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem](https://mlsanity.com/paper/2206.04119/) (2022)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [SE(3) diffusion model with application to protein backbone generation](https://mlsanity.com/paper/2302.02277/) (2023)
- [One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale](https://mlsanity.com/paper/2303.06555/) (2023)
- [ProGen2: Exploring the Boundaries of Protein Language Models](https://mlsanity.com/paper/2206.13517/) (2022)
- [A Reparameterized Discrete Diffusion Model for Text Generation](https://mlsanity.com/paper/2302.05737/) (2023)
- [HART: Efficient Visual Generation with Hybrid Autoregressive Transformer](https://mlsanity.com/paper/2410.10812/) (2024)
- [Protein structure generation via folding diffusion](https://mlsanity.com/paper/2209.15611/) (2022)
- [Elucidating the Exposure Bias in Diffusion Models](https://mlsanity.com/paper/2308.15321/) (2023)
- [Diffusion Language Models Are Versatile Protein Learners](https://mlsanity.com/paper/2402.18567/) (2024)
- [Input Perturbation Reduces Exposure Bias in Diffusion Models](https://mlsanity.com/paper/2301.11706/) (2023)
- [Iterative Refinement Graph Neural Network for Antibody Sequence-Structure Co-design](https://mlsanity.com/paper/2110.04624/) (2021)
- [PiFold: Toward effective and efficient protein inverse folding](https://mlsanity.com/paper/2209.12643/) (2022)
- [Conditional Antibody Design as 3D Equivariant Graph Translation](https://mlsanity.com/paper/2208.06073/) (2022)
- [Alleviating Exposure Bias in Diffusion Models through Sampling with Shifted Time Steps](https://mlsanity.com/paper/2305.15583/) (2023)
- [Structure-informed Language Models Are Protein Designers](https://mlsanity.com/paper/2302.01649/) (2023)
- [Generating Novel, Designable, and Diverse Protein Structures by Equivariantly Diffusing Oriented Residue Clouds](https://mlsanity.com/paper/2301.12485/) (2023)

## Built on by (most cited first, 30 of 64)

- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation](https://mlsanity.com/paper/2510.06303/) (2025)
- [Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review](https://mlsanity.com/paper/2501.09685/) (2025)
- [Path Planning for Masked Diffusion Model Sampling](https://mlsanity.com/paper/2502.03540/) (2025)
- [A Survey on Diffusion Language Models](https://mlsanity.com/paper/2508.10875/) (2025)
- [Discrete Diffusion in Large Language and Multimodal Models: A Survey](https://mlsanity.com/paper/2506.13759/) (2025)
- [Attention Is All You Need for KV Cache in Diffusion LLMs](https://mlsanity.com/paper/2510.14973/) (2025)
- [La-Proteina: Atomistic Protein Generation via Partially Latent Flow Matching](https://mlsanity.com/paper/2507.09466/) (2025)
- [Reward-Guided Iterative Refinement in Diffusion Models at Test-Time with Applications to Protein and DNA Design](https://mlsanity.com/paper/2502.14944/) (2025)
- [α-Flow: A Unified Framework for Continuous-State Discrete Flow Matching Models](https://mlsanity.com/paper/2504.10283/) (2025)
- [Planner Aware Path Learning in Diffusion Language Models Training](https://mlsanity.com/paper/2509.23405/) (2025)
- [Elucidating the Design Space of Multimodal Protein Language Models](https://mlsanity.com/paper/2504.11454/) (2025)
- [Self-Speculative Masked Diffusions](https://mlsanity.com/paper/2510.03929/) (2025)
- [An All-Atom Generative Model for Designing Protein Complexes](https://mlsanity.com/paper/2504.13075/) (2025)
- [mCLM: A Modular Chemical Language Model that Generates Functional and Makeable Molecules](https://mlsanity.com/paper/2505.12565/) (2025)
- [Computational Protein Science in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2501.10282/) (2025)
- [ProtInvTree: Deliberate Protein Inverse Folding with Reward-guided Tree Search](https://mlsanity.com/paper/2506.00925/) (2025)
- [Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings](https://mlsanity.com/paper/2509.22925/) (2025)
- [PRISM: Enhancing Protein Inverse Folding through Fine-Grained Retrieval on Structure-Sequence Multimodal Representations](https://mlsanity.com/paper/2510.11750/) (2025)
- [Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generator](https://mlsanity.com/paper/2503.15457/) (2025)
- [Protein Design with Dynamic Protein Vocabulary](https://mlsanity.com/paper/2505.18966/) (2025)
- [PDFBench: A Benchmark for De novo Protein Design from Function](https://mlsanity.com/paper/2505.20346/) (2025)
- [SafeProtein: Red-Teaming Framework and Benchmark for Protein Foundation Models](https://mlsanity.com/paper/2509.03487/) (2025)
- [Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusion](https://mlsanity.com/paper/2605.23346/) (2026)
- [Flow Autoencoders are Effective Protein Tokenizers](https://mlsanity.com/paper/2510.00351/) (2025)
- [PartDiffuser: Part-wise 3D Mesh Generation via Discrete Diffusion](https://mlsanity.com/paper/2511.18801/) (2025)
- [HD-Prot: A Protein Language Model for Joint Sequence-Structure Modeling with Continuous Structure Tokens](https://mlsanity.com/paper/2512.15133/) (2025)
- [Masked Diffusion Models as Energy Minimization](https://mlsanity.com/paper/2509.13866/) (2025)
- [ProteinAE: Protein Diffusion Autoencoders for Structure Encoding](https://mlsanity.com/paper/2510.10634/) (2025)
- [DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Discrete Diffusion Models](https://mlsanity.com/paper/2604.24357/) (2026)

## Abstract (arXiv)

Proteins are essential macromolecules defined by their amino acid sequences, which determine their three-dimensional structures and, consequently, their functions in all living organisms. Therefore, generative protein modeling necessitates a multimodal approach to simultaneously model, understand, and generate both sequences and structures. However, existing methods typically use separate models for each modality, limiting their ability to capture the intricate relationships between sequence and structure. This results in suboptimal performance in tasks that requires joint understanding and generation of both modalities. In this paper, we introduce DPLM-2, a multimodal protein foundation model that extends discrete diffusion protein language model (DPLM) to accommodate both sequences and structures. To enable structural learning with the language model, 3D coordinates are converted to discrete tokens using a lookup-free quantization-based tokenizer. By training on both experimental and high-quality synthetic structures, DPLM-2 learns the joint distribution of sequence and structure, as well as their marginals and conditionals. We also implement an efficient warm-up strategy to exploit the connection between large-scale evolutionary data and structural inductive biases from pre-trained sequence-based protein language models. Empirical evaluation shows that DPLM-2 can simultaneously generate highly compatible amino acid sequences and their corresponding 3D structures eliminating the need for a two-stage generation approach. Moreover, DPLM-2 demonstrates competitive performance in various conditional generation tasks, including folding, inverse folding, and scaffolding with multimodal motif inputs, as well as providing structure-aware representations for predictive tasks.