mlsanityData to 14 Oct 2026

Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking

arXiv 2505.18495 ↗First posted 24 May 2025cs.LGcited by 20

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Masked diffusion models (MDM) are powerful generative models for discrete data that generate samples by progressively unmasking tokens in a sequence. Each token can take one of two states: masked or unmasked. We observe that token sequences often remain unchanged between consecutive sampling steps; consequently, the model repeatedly processes identical inputs, leading to redundant computation. To address this inefficiency, we propose the Partial masking scheme (Prime), which augments MDM by allowing tokens to take intermediate states interpolated between the masked and unmasked states. This design enables the model to make predictions based on partially observed token information, and facilitates a fine-grained denoising process. We derive a variational training objective and introduce a simple architectural design to accommodate intermediate-state inputs. Our method demonstrates superior performance across a diverse set of generative modeling tasks. On text data, it achieves a perplexity of 15.36 on OpenWebText, outperforming previous MDM (21.52), autoregressive models (17.54), and their hybrid variants (17.58), without relying on an autoregressive formulation. On image data, it attains competitive FID scores of 3.26 on CIFAR-10 and 6.98 on ImageNet-32, comparable to leading continuous generative models.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 51 of its 68 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

51 of 68 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  2. 2025Remasking Discrete Diffusion Models with Inference-Time Scaling1 Mar
  3. 2025Large Language Diffusion Models14 Feb
  4. 2024Energy-Based Diffusion Language Models for Text Generation28 Oct
  5. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time28 Oct
  6. 2024Discrete Copula Diffusion2 Oct
  7. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  8. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  9. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  10. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  11. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  12. 2023Likelihood-Based Diffusion Language Models30 May
1 more
  1. 2022Continuous diffusion for categorical data28 Nov

Other references · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Scalable Diffusion Models with Transformerscited by 6,054
  7. 2022Flow Matching for Generative Modelingcited by 5,702
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  12. 2016Density estimation using Real NVPcited by 2,042
26 more
  1. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  2. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  3. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  4. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
  5. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  6. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  8. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  9. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  10. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  11. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  12. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  13. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  14. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
  15. 2024Discrete Flow Matchingcited by 317
  16. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  17. 2018A Note on the Inception Scorecited by 226
  18. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 129
  19. 2021Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimationcited by 97
  20. 2024Informed Correctors for Discrete Diffusion Modelscited by 57
  21. 2024Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspectivecited by 57
  22. 2023Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEscited by 54
  23. 2024Distillation of Discrete Diffusion through Dimensional Correlationscited by 48
  24. 2019Self-supervised GAN: Analysis and Improvement with Multi-class Minimax Gamecited by 25
  25. 2023Neural Diffusion Modelscited by 21
  26. 2022On Investigating the Conservative Property of Score-Based Generative Modelscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 12.

20 papers.

Sort:

By area:

  1. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  2. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 34
  3. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasonercited by 32
  4. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 20
  5. 2025Soft-Masked Diffusion Language Modelscited by 16
  6. 2025Corrective Diffusion Language Modelscited by 11
  7. 2025Enabling Approximate Joint Sampling in Diffusion LMscited by 10
  8. 2025Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wallcited by 8
  9. 2026Accelerating Discrete Diffusion Models with Parallel-In-Time Samplingcited by 4
  10. 2026Flow Matching for Count Datacited by 3
  11. 2026Lattice Deduction Transformerscited by 3
  12. 2026Recursive Scaling in Masked Diffusion Modelscited by 2
8 more
  1. 2026Sticky Jump Diffusions: A Unifying View of Masked, Continuous, and Hybrid Diffusioncited by 2
  2. 2026Rethinking Token Prediction: Tree-Structured Diffusion Language Modelcited by 1
  3. 2026Binomial flows: Denoising and flow matching for discrete ordinal datacited by 1
  4. 2026Looped Diffusion Language Modelscited by 1
  5. 2026Multi-Mask Diffusion Language Models for Few-Step Generationcited by 1
  6. 2026MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Scaling of Diffusion Language Modelscited by 0
  7. 2026LaDA-Band: Language Diffusion Models for Vocal-to-Accompaniment Generationcited by 0
  8. 2026Jumping up and down: Denoiser diffusion models for discrete ordinal datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Transformer-Based Masked Diffusion Models Provably Learn Distributions with Latent Structure
  2. 2026Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training
  3. 2026Learn from Your Mistakes: Self-Correcting Masked Diffusion Models
  4. 2025Error Bounds and Optimal Schedules for Masked Diffusions with Factorized Approximations
  5. 2025Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generator
  6. 2025Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing
  7. 2025Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusion
  8. 2025Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models