mlsanityData to 7 Oct 2026

Simplified and Generalized Masked Diffusion for Discrete Data

arXiv 2406.04329 ↗First posted 6 Jun 2024cs.LG stat.MLcited by 606

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Masked (or absorbing) diffusion is actively explored as an alternative to autoregressive models for generative modeling of discrete data. However, existing work in this area has been hindered by unnecessarily complex model formulations and unclear relationships between different perspectives, leading to suboptimal parameterization, training objectives, and ad hoc adjustments to counteract these issues. In this work, we aim to provide a simple and general framework that unlocks the full potential of masked diffusion models. We show that the continuous-time variational objective of masked diffusion models is a simple weighted integral of cross-entropy losses. Our framework also enables training generalized masked diffusion models with state-dependent masking schedules. When evaluated by perplexity, our models trained on OpenWebText surpass prior diffusion language models at GPT-2 scale and demonstrate superior performance on 4 out of 5 zero-shot language modeling tasks. Furthermore, our models vastly outperform previous discrete diffusion models on pixel-level image modeling, achieving 2.75 (CIFAR-10) and 3.40 (ImageNet 64x64) bits per dimension that are better than autoregressive models of similar sizes. Our code is available at https://github.com/google-deepmind/md4.

Key facts. Cited by 606 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 62 of its 79 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

62 of 79 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Transfer Learning for Text Diffusion Models30 Jan
  2. 2023DiSK: A Diffusion Model for Structured Knowledge8 Dec
  3. 2023Likelihood-Based Diffusion Language Models30 May
  4. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  5. 2022Latent Diffusion for Language Generation19 Dec
  6. 2022Continuous diffusion for categorical data28 Nov
  7. 2021Autoregressive Diffusion Models5 Oct

Other references · 55

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
38 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2022Video Diffusion Modelscited by 2,280
  3. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  5. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  6. 2021Variational Diffusion Modelscited by 1,145
  7. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  8. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  11. 2016Pixel Recurrent Neural Networkscited by 814
  12. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  13. 2018Image Transformercited by 770
  14. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  15. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  16. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 503
  17. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  18. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  19. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  20. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  21. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  22. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  23. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  24. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 294
  25. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  26. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 167
  27. 2024Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Modelscited by 159
  28. 2023Protein Design with Guided Discrete Diffusioncited by 154
  29. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137
  30. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 124
  31. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  32. 2023Dirichlet Diffusion Score Model for Biological Sequence Generationcited by 73
  33. 2019Discrete Flows: Invertible Generative Models of Discrete Datacited by 68
  34. 2013A Deep and Tractable Density Estimatorcited by 67
  35. 2019Latent Normalizing Flows for Discrete Sequencescited by 66
  36. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  37. 2023Bayesian Flow Networkscited by 62
  38. 2022From Denoising Diffusions to Denoising Markov Modelscited by 58

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 240; 2026: 331.

606 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 872
  2. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  3. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  4. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  5. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  6. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  7. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  8. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  9. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 204
  10. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 201
  11. 2024Scaling up Masked Diffusion Models on Textcited by 195
  12. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
38 more
  1. 2025Remasking Discrete Diffusion Models with Inference-Time Scalingcited by 181
  2. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  3. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  4. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  5. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 148
  6. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  7. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  8. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 128
  9. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
  10. 2025The Diffusion Dualitycited by 124
  11. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 118
  12. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  13. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  14. 2024Energy-Based Diffusion Language Models for Text Generationcited by 86
  15. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
  16. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
  17. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  18. 2025Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistencycited by 81
  19. 2025Path Planning for Masked Diffusion Model Samplingcited by 79
  20. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  21. 2024Dual Diffusion for Unified Image Generation and Understandingcited by 76
  22. 2025A Survey on Diffusion Language Modelscited by 75
  23. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Timecited by 73
  24. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 72
  25. 2025Generalized Interpolating Discrete Diffusioncited by 68
  26. 2025Diffusion Language Models Know the Answer Before Decodingcited by 68
  27. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  28. 2024Think While You Generate: Discrete Diffusion with Planned Denoisingcited by 61
  29. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  30. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 61
  31. 2024TabDiff: a Mixed-type Diffusion Model for Tabular Data Generationcited by 60
  32. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 59
  33. 2024Informed Correctors for Discrete Diffusion Modelscited by 57
  34. 2024Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspectivecited by 54
  35. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
  36. 2024Discrete Copula Diffusioncited by 53
  37. 2025Edit Flows: Flow Matching with Edit Operationscited by 53
  38. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 52

50 most cited of 606 All 606 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Masked Diffusion Models Are Fast Distribution Learners
  2. 2025Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generator
  3. 2023Diffusion Models as Masked Autoencoders
  4. 2025Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models
  5. 2023Fast Training of Diffusion Models with Masked Transformers
  6. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noise
  7. 2023Masked Diffusion as Self-supervised Representation Learner
  8. 2022Diffusion Models in Vision: A Survey