mlsanityData to 7 Oct 2026

Diffusion Language Models Are Versatile Protein Learners

arXiv 2402.18567 ↗First posted 28 Feb 2024cs.LG q-bio.BMcited by 93

On the map: Diffusion models → Flow matching, time series and science

Abstract (arXiv)

This paper introduces diffusion protein language model (DPLM), a versatile protein language model that demonstrates strong generative and predictive capabilities for protein sequences. We first pre-train scalable DPLMs from evolutionary-scale protein sequences within a generative self-supervised discrete diffusion probabilistic framework, which generalizes language modeling for proteins in a principled way. After pre-training, DPLM exhibits the ability to generate structurally plausible, novel, and diverse protein sequences for unconditional generation. We further demonstrate the proposed diffusion generative pre-training makes DPLM possess a better understanding of proteins, making it a superior representation learner, which can be fine-tuned for various predictive tasks, comparing favorably to ESM2 (Lin et al., 2022). Moreover, DPLM can be tailored for various needs, which showcases its prowess of conditional generation in several ways: (1) conditioning on partial peptide sequences, e.g., generating scaffolds for functional motifs with high success rate; (2) incorporating other modalities as conditioner, e.g., structure-conditioned generation for inverse folding; and (3) steering sequence generation towards desired properties, e.g., satisfying specified secondary structures, through a plug-and-play classifier guidance. Code is released at urlhttps://github.com/bytedance/dplm.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in Flow matching, time series and science, one of 14 directions in Diffusion models. Builds on 35 of its 131 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

35 of 131 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023SE(3) diffusion model with application to protein backbone generation5 Feb
  2. 2022Protein structure generation via folding diffusion30 Sep
  3. 2022Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem8 Jun

Other references · 32

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  11. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 439
  12. 2023Scaling Data-Constrained Language Modelscited by 391
20 more
  1. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  2. 2022Continuous diffusion for categorical datacited by 202
  3. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Controlcited by 191
  4. 2022ProGen2: Exploring the Boundaries of Protein Language Modelscited by 156
  5. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  6. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generationcited by 113
  7. 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learningcited by 89
  8. 2022PiFold: Toward effective and efficient protein inverse foldingcited by 82
  9. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformerscited by 63
  10. 2023Generating Novel, Designable, and Diverse Protein Structures by Equivariantly Diffusing Oriented Residue Cloudscited by 53
  11. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noisescited by 50
  12. 2022Empowering Diffusion Models on the Embedding Space for Text Generationcited by 40
  13. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuningcited by 39
  14. 2021Modeling Protein Using Large-scale Pretrain Language Modelcited by 21
  15. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noisecited by 21
  16. 2021Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Modelcited by 20
  17. 2022Reprogramming Pretrained Language Models for Antibody Sequence Infillingcited by 20
  18. 2020Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Modelscited by 12
  19. 2021Adversarial Contrastive Pre-training for Protein Sequencescited by 8
  20. 2022Diffusion Glancing Transformer for Parallel Sequence to Sequence Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 45; 2026: 35.

93 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 872
  2. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  3. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 124
  4. 2025Path Planning for Masked Diffusion Model Samplingcited by 79
  5. 2025A Survey on Diffusion Language Modelscited by 75
  6. 2024DPLM-2: A Multimodal Diffusion Protein Language Modelcited by 59
  7. 2024On Statistical Rates and Provably Efficient Criteria of Latent Diffusion Transformers (DiTs)cited by 39
  8. 2024PepTune: De Novo Generation of Therapeutic Peptides with Multi-Objective-Guided Discrete Diffusioncited by 35
  9. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  10. 2024Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Predictioncited by 28
  11. 2025La-Proteina: Atomistic Protein Generation via Partially Latent Flow Matchingcited by 26
  12. 2024ProteinBench: A Holistic Evaluation of Protein Foundation Modelscited by 23
38 more
  1. 2025MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Controlcited by 20
  2. 2025Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spacescited by 19
  3. 2025α-Flow: A Unified Framework for Continuous-State Discrete Flow Matching Modelscited by 17
  4. 2025Ultra-Fast Language Generation via Discrete Diffusion Divergence Instructcited by 15
  5. 2025Offline Model-Based Optimization: Comprehensive Reviewcited by 14
  6. 2025Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusioncited by 14
  7. 2025Planner Aware Path Learning in Diffusion Language Models Trainingcited by 14
  8. 2025TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusioncited by 13
  9. 2024Diffusion on language model encodings for protein sequence generationcited by 11
  10. 2025Self-Speculative Masked Diffusionscited by 11
  11. 2024Diffusion Model with Representation Alignment for Protein Inverse Foldingcited by 10
  12. 2025Elucidating the Design Space of Multimodal Protein Language Modelscited by 10
  13. 2024Learning the Language of Protein Structurecited by 9
  14. 2025An All-Atom Generative Model for Designing Protein Complexescited by 9
  15. 2026Discrete Feynman-Kac Correctorscited by 9
  16. 2025Diffusion Generative Recommendation with Continuous Tokenscited by 8
  17. 2025CFP-Gen: Combinatorial Functional Protein Generation via Diffusion Language Modelscited by 8
  18. 2025All-atom inverse protein folding through discrete flow matchingcited by 8
  19. 2026The data geometry of masking diffusion: Certified-optimal schedules via unmasking growth complexitycited by 8
  20. 2025Steering Generative Models with Experimental Data for Protein Fitness Optimizationcited by 7
  21. 2025Computational Protein Science in the Era of Large Language Models (LLMs)cited by 6
  22. 2025A Diffusion Model to Shrink Proteins While Maintaining Their Functioncited by 6
  23. 2026D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generationcited by 6
  24. 2025Protein Design with Dynamic Protein Vocabularycited by 5
  25. 2025PDFBench: A Benchmark for De novo Protein Design from Functioncited by 5
  26. 2025ProtInvTree: Deliberate Protein Inverse Folding with Reward-guided Tree Searchcited by 5
  27. 2025AReUReDi: Annealed Rectified Updates for Refining Discrete Flows with Multi-Objective Guidancecited by 5
  28. 2025PRISM: Enhancing Protein Inverse Folding through Fine-Grained Retrieval on Structure-Sequence Multimodal Representationscited by 5
  29. 2024CryoFM: A Flow-based Foundation Model for Cryo-EM Densitiescited by 4
  30. 2025ProtFlow: Fast Protein Sequence Design via Flow Matching on Compressed Protein Language Model Embeddingscited by 4
  31. 2025HD-Prot: A Protein Language Model for Joint Sequence-Structure Modeling with Continuous Structure Tokenscited by 4
  32. 2026Controlling Repetition in Protein Language Modelscited by 4
  33. 2024Function-Guided Conditional Generation Using Protein Language Models with Adapterscited by 3
  34. 2025VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RLcited by 3
  35. 2025ShortListing Model: A Streamlined SimplexDiffusion for Discrete Variable Generationcited by 3
  36. 2025Masked Diffusion Models as Energy Minimizationcited by 3
  37. 2026MaDiS: Taming Masked Diffusion Language Models for Sign Language Generationcited by 3
  38. 2026Primal-Dual Guided Decoding for Constrained Discrete Diffusioncited by 3

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference
  2. 2024TaxDiff: Taxonomic-Guided Diffusion Model for Protein Sequence Generation
  3. 2024Structure Language Models for Protein Conformation Generation
  4. 2024TourSynbio: A Multi-Modal Large Model and Agent Framework to Bridge Text and Protein Sequences for Protein Engineering
  5. 2025From thermodynamics to protein design: Diffusion models for biomolecule generation towards autonomous protein engineering
  6. 2025Demystify Protein Generation with Hierarchical Conditional Diffusion Models
  7. 2022Protein Structure and Sequence Generation with Equivariant Denoising Diffusion Probabilistic Models
  8. 2023PoET: A generative model of protein families as sequences-of-sequences