mlsanityData to 14 Oct 2026

Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling

arXiv 2409.02908 ↗First posted 4 Sep 2024cs.LG cs.AI cs.CLcited by 208

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Masked diffusion models (MDMs) have emerged as a popular research topic for generative modeling of discrete data, thanks to their superior performance over other discrete diffusion models, and are rivaling the auto-regressive models (ARMs) for language modeling tasks. The recent effort in simplifying the masked diffusion framework further leads to alignment with continuous-space diffusion models and more principled training and sampling recipes. In this paper, however, we reveal that both training and sampling of MDMs are theoretically free from the time variable, arguably the key signature of diffusion models, and are instead equivalent to masked models. The connection on the sampling aspect is drawn by our proposed first-hitting sampler (FHS). Specifically, we show that the FHS is theoretically equivalent to MDMs' original generation process while significantly alleviating the time-consuming categorical sampling and achieving a 20× speedup. In addition, our investigation raises doubts about whether MDMs can truly beat ARMs in text generation. We identify, for the first time, an underlying numerical issue, even with the commonly used 32-bit floating-point precision, which results in inaccurate categorical sampling. We show that it lowers the effective temperature both theoretically and empirically, and the resulting decrease in token diversity makes previous evaluations, which assess the generation quality solely through the incomplete generative perplexity metric, somewhat unfair.

Key facts. Cited by 208 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 49 of its 63 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

49 of 63 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  2. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  3. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  4. 2021Autoregressive Diffusion Models5 Oct

Other references · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2020Denoising Diffusion Implicit Modelscited by 9,781
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
33 more
  1. 2022Scalable Diffusion Models with Transformerscited by 6,054
  2. 2022Flow Matching for Generative Modelingcited by 5,702
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  7. 2023Mistral 7Bcited by 3,164
  8. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  10. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  12. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  13. 2021Variational Diffusion Modelscited by 1,153
  14. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  15. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  16. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  17. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  18. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  19. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 532
  20. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  21. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
  22. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  23. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 257
  24. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 169
  25. 2023DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statisticscited by 161
  26. 2024Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Modelscited by 160
  27. 2022Concrete Score Matching: Generalized Score Matching for Discrete Datacited by 148
  28. 2022Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matchingcited by 80
  29. 2013A Deep and Tractable Density Estimatorcited by 68
  30. 2023Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEscited by 54
  31. 2024Diffusion Bridge Implicit Modelscited by 49
  32. 2024Convergence Analysis of Discrete Diffusion Model: Exact Implementation through Uniformizationcited by 44
  33. 2023SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Modelscited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 87; 2026: 110.

208 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 912
  2. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  3. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  4. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  5. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 214
  6. 2024Scaling up Masked Diffusion Models on Textcited by 199
  7. 2025Remasking Discrete Diffusion Models with Inference-Time Scalingcited by 189
  8. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 160
  9. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  10. 2025The Diffusion Dualitycited by 135
  11. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 131
  12. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
38 more
  1. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 114
  2. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  3. 2025Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistencycited by 86
  4. 2025Path Planning for Masked Diffusion Model Samplingcited by 80
  5. 2025Generalized Interpolating Discrete Diffusioncited by 70
  6. 2025Diffusion Language Models Know the Answer Before Decodingcited by 70
  7. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 65
  8. 2024Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspectivecited by 57
  9. 2025Edit Flows: Flow Matching with Edit Operationscited by 55
  10. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Modelscited by 54
  11. 2025Any-Order Flexible Length Masked Diffusioncited by 51
  12. 2025Unified Multimodal Discrete Diffusioncited by 49
  13. 2025Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithmscited by 47
  14. 2025Theoretical Benefit and Limitation of Diffusion Language Modelcited by 47
  15. 2025KLASS: KL-Guided Fast Inference in Masked Diffusion Modelscited by 47
  16. 2025Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principlescited by 45
  17. 2025Fine-Tuning Masked Diffusion for Provable Self-Correctioncited by 39
  18. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  19. 2024PepTune: De Novo Generation of Therapeutic Peptides with Multi-Objective-Guided Discrete Diffusioncited by 37
  20. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 34
  21. 2025CANDI: Hybrid Discrete-Continuous Diffusion Modelscited by 34
  22. 2024Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Predictioncited by 32
  23. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  24. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  25. 2025Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decodingcited by 27
  26. 2026Scaling Beyond Masked Diffusion Language Modelscited by 27
  27. 2025Latent Thought Models with Variational Bayes Inference-Time Computationcited by 25
  28. 2026Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Modelscited by 25
  29. 2025Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminatorcited by 23
  30. 2025From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Modelscited by 23
  31. 2025Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policiescited by 22
  32. 2025MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Controlcited by 21
  33. 2025Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Maskingcited by 20
  34. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generationcited by 20
  35. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 20
  36. 2026Continuous Diffusion Scales Competitively with Discrete Diffusion for Languagecited by 19
  37. 2026Confidence-Based Decoding is Provably Efficient for Diffusion Language Modelscited by 18
  38. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 18

50 most cited of 208 All 208 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimal Inference Schedules for Masked Diffusion Models
  2. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noise
  3. 2026Infinite Mask Diffusion for Few-Step Distillation
  4. 2025Improving Text Style Transfer using Masked Diffusion Language Models with Inference-time Scaling
  5. 2025Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing
  6. 2026Representation-based Masked Diffusion Model
  7. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models
  8. 2025Plan for Speed: Dilated Scheduling for Masked Diffusion Language Models