mlsanityData to 14 Oct 2026

DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models

arXiv 2211.15029 ↗First posted 28 Nov 2022cs.CL cs.AI cs.LGcited by 214

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

We present DiffusionBERT, a new generative masked language model based on discrete diffusion models. Diffusion models and many pre-trained language models have a shared training objective, i.e., denoising, making it possible to combine the two powerful models and enjoy the best of both worlds. On the one hand, diffusion models offer a promising training strategy that helps improve the generation quality. On the other hand, pre-trained denoising language models (e.g., BERT) can be used as a good initialization that accelerates convergence. We explore training BERT to learn the reverse process of a discrete diffusion process with an absorbing state and elucidate several designs to improve it. First, we propose a new noise schedule for the forward diffusion process that controls the degree of noise added at each step based on the information of each token. Second, we investigate several designs of incorporating the time step into BERT. Experiments on unconditional text generation demonstrate that DiffusionBERT achieves significant improvement over existing diffusion models for text (e.g., D3PM and Diffusion-LM) and previous generative masked language models in terms of perplexity and BLEU score.

Key facts. Cited by 214 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 29 of its 42 references that are in the map. First posted on arXiv 28 Nov 2022.

Builds on

29 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct
  2. 2021Autoregressive Diffusion Models5 Oct

Other references · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
15 more
  1. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  2. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  4. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  5. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  6. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  7. 2017Non-Autoregressive Neural Machine Translationcited by 463
  8. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  9. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  10. 2019Levenshtein Transformercited by 199
  11. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 185
  12. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  13. 2018Language GANs Falling Shortcited by 97
  14. 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
  15. 2017Quasar: Datasets for Question Answering by Search and Readingcited by 50

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 26; 2024: 35; 2025: 75; 2026: 75.

214 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 912
  2. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  3. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  4. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  5. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  6. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  7. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  8. 2024Discrete Flow Matchingcited by 317
  9. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  10. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274
  11. 2023DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimizationcited by 221
  12. 2024Scaling up Masked Diffusion Models on Textcited by 199
38 more
  1. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  2. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  3. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 143
  4. 2025The Diffusion Dualitycited by 135
  5. 2022Latent Diffusion for Language Generationcited by 115
  6. 2023DiffuRec: A Diffusion Model for Sequential Recommendationcited by 93
  7. 2024A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Datacited by 93
  8. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  9. 2025A Survey on Diffusion Language Modelscited by 78
  10. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformerscited by 64
  11. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  12. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 57
  13. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noisescited by 50
  14. 2024How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Frameworkcited by 48
  15. 2025Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithmscited by 47
  16. 2025Theoretical Benefit and Limitation of Diffusion Language Modelcited by 47
  17. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  18. 2023BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesiscited by 45
  19. 2026ELF: Embedded Language Flowscited by 45
  20. 2023Diffusion Models for Non-autoregressive Text Generation: A Surveycited by 44
  21. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 42
  22. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 40
  23. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuningcited by 39
  24. 2023Diffusion Augmentation for Sequential Recommendationcited by 38
  25. 2023DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Modelscited by 36
  26. 2026Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generationcited by 36
  27. 2026LangFlow: Continuous Diffusion Rivals Discrete in Language Modelingcited by 36
  28. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 34
  29. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 34
  30. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 33
  31. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  32. 2024Text-Guided Molecule Generation with Diffusion Language Modelcited by 31
  33. 2025Continuous Diffusion Model for Language Modelingcited by 31
  34. 2025Self Speculative Decoding for Diffusion Large Language Modelscited by 30
  35. 2024Diff-RNTraj: A Structure-aware Diffusion Model for Road Network-constrained Trajectory Generationcited by 29
  36. 2025Jailbreaking Large Language Diffusion Models: Revealing Hidden Safety Flaws in Diffusion-Based Text Generationcited by 26
  37. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 24
  38. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Modelscited by 23

50 most cited of 214 All 214 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Discrete Diffusion Models for Language Generation
  2. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling
  3. 2026Diffusion Language Models: An Experimental Analysis
  4. 2025Simple Denoising Diffusion Language Models
  5. 2025Improving Text Style Transfer using Masked Diffusion Language Models with Inference-time Scaling
  6. 2026How to Train Your Latent Diffusion Language Model Jointly With the Latent Space
  7. 2026Unifying Masked Diffusion Models with Various Generation Orders and Beyond
  8. 2022Empowering Diffusion Models on the Embedding Space for Text Generation