mlsanityData to 7 Oct 2026

Simple and Effective Masked Diffusion Language Models

arXiv 2406.07524 ↗First posted 11 Jun 2024cs.CL cs.AI cs.LGcited by 800

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has a simple form -- it is a mixture of classical masked language modeling losses -- and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We provide the code, along with a blog post and video tutorial on the project page: https://s-sahoo.com/mdlm

Key facts. Cited by 800 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 54 of its 73 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

54 of 73 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  2. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  3. 2023Likelihood-Based Diffusion Language Models30 May
  4. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  5. 2022Latent Diffusion for Language Generation19 Dec
  6. 2022Continuous diffusion for categorical data28 Nov
  7. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  8. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov
  9. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct

Other references · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
33 more
  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  4. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  5. 2021Variational Diffusion Modelscited by 1,145
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  8. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 503
  11. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  12. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  13. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  14. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  15. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 294
  16. 2023DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimizationcited by 216
  17. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 187
  18. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  19. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 167
  20. 2022Concrete Score Matching: Generalized Score Matching for Discrete Datacited by 144
  21. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
  22. 2023Dirichlet Diffusion Score Model for Biological Sequence Generationcited by 73
  23. 2022From Denoising Diffusions to Denoising Markov Modelscited by 58
  24. 2023Diffusion Models With Learned Adaptive Noisecited by 50
  25. 2023InfoDiffusion: Representation Learning Using Information Maximizing Diffusion Modelscited by 48
  26. 2022Backpropagation through Combinatorial Algorithms: Identity with Projection Workscited by 29
  27. 2020Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Ordercited by 28
  28. 2021OmniNet: Omnidirectional Representations from Transformerscited by 23
  29. 2021Discovering Non-monotonic Autoregressive Orderings with Variational Inferencecited by 15
  30. 2021Autoregressive Quantile Flows for Predictive Uncertainty Estimationcited by 15
  31. 2022Deep Multi-Modal Structural Equations For Causal Effect Estimation With Unstructured Proxiescited by 13
  32. 2022Semi-Autoregressive Energy Flows: Exploring Likelihood-Free Training of Normalizing Flowscited by 9
  33. 2022Semi-Parametric Inducing Point Networks and Neural Processescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 269; 2026: 498.

800 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 872
  2. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  3. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  4. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  5. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  6. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  7. 2024Diffusion Policy Policy Optimizationcited by 255
  8. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  9. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  10. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 204
  11. 2024Scaling up Masked Diffusion Models on Textcited by 195
  12. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
38 more
  1. 2025Remasking Discrete Diffusion Models with Inference-Time Scalingcited by 181
  2. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  3. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  4. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  5. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 148
  6. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  7. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  8. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  9. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 128
  10. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
  11. 2025The Diffusion Dualitycited by 124
  12. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 118
  13. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  14. 2025Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Expertscited by 85
  15. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  16. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
  17. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
  18. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  19. 2025Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistencycited by 81
  20. 2025Path Planning for Masked Diffusion Model Samplingcited by 79
  21. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  22. 2024Dual Diffusion for Unified Image Generation and Understandingcited by 76
  23. 2025A Survey on Diffusion Language Modelscited by 75
  24. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Timecited by 73
  25. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 72
  26. 2025Generalized Interpolating Discrete Diffusioncited by 68
  27. 2025Diffusion Language Models Know the Answer Before Decodingcited by 68
  28. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  29. 2024Think While You Generate: Discrete Diffusion with Planned Denoisingcited by 61
  30. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  31. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 61
  32. 2024TabDiff: a Mixed-type Diffusion Model for Tabular Data Generationcited by 60
  33. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 59
  34. 2024Informed Correctors for Discrete Diffusion Modelscited by 57
  35. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  36. 2024Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspectivecited by 54
  37. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
  38. 2024Discrete Copula Diffusioncited by 53

50 most cited of 800 All 800 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noise
  2. 2025Why mask diffusion does not work
  3. 2025Discrete Diffusion Models for Language Generation
  4. 2025Theoretical Benefit and Limitation of Diffusion Language Model
  5. 2025DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models
  6. 2026The Design Space of Tri-Modal Masked Diffusion Models
  7. 2026Diffutron: A Masked Diffusion Language Model for Turkish Language
  8. 2026LESS Is More: Mutual-Stability Sampling for Diffusion Language Models