mlsanityData to 7 Oct 2026

Generation Order and Parallel Decoding in Masked Diffusion Models: An Information-Theoretic Perspective

arXiv 2602.00286 ↗First posted 30 Jan 2026cs.LGcited by 9

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

Masked Diffusion Models (MDMs) significantly accelerate inference by trading off sequential determinism. However, the theoretical mechanisms governing generation order and the risks inherent in parallelization remain under-explored. In this work, we provide a unified information-theoretic framework to decouple and analyze two fundamental sources of failure: order sensitivity and parallelization bias. Our analysis yields three key insights: (1) The benefits of Easy-First decoding (prioritizing low-entropy tokens) are magnified as model error increases; (2) factorized parallel decoding introduces intrinsic sampling errors that can lead to arbitrary large Reverse KL divergence, capturing "incoherence" failures that standard Forward KL metrics overlook; and (3) while verification can eliminate sampling error, it incurs an exponential cost governed by the total correlation within a block. Conversely, heuristics like remasking, though computationally efficient, cannot guarantee distributional correctness. Experiments on a controlled Block-HMM and large-scale MDMs (LLaDA) for arithmetic reasoning validate our theoretical framework.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 20 of its 22 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

20 of 22 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Optimal Inference Schedules for Masked Diffusion Models6 Nov
  2. 2025Large Language Diffusion Models14 Feb
  3. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions10 Feb
  4. 2025Path Planning for Masked Diffusion Model Sampling5 Feb
  5. 2024Simple and Effective Masked Diffusion Language Models11 Jun

Other references · 15

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  7. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  8. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  9. 2017Non-Autoregressive Neural Machine Translationcited by 457
  10. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  11. 2024Discrete Flow Matchingcited by 301
  12. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 294
3 more
  1. 2025Edit Flows: Flow Matching with Edit Operationscited by 53
  2. 2025ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMscited by 42
  3. 2024Speculative Decoding with CTC-based Draft Model for LLM Inference Accelerationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Limits of Confidence in Diffusioncited by 2
  2. 2026Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Modelscited by 1
  3. 2026Supportive Token Revealing for Fast Diffusion Language Model Decodingcited by 1
  4. 2026Tensor-Train Joint Modeling for Few-Step Discrete Diffusioncited by 1
  5. 2026Conditional Total Correlation and the Serial Depth of Adaptive Parallel Samplingcited by 1
  6. 2026Decoding in Order-Agnostic Language Models: Chain-Rule Deviation and Uniform Spreadingcited by 0
  7. 2026Mean-Field Parallel Decoding for Discrete Diffusion Language Modelscited by 0
  8. 2026Unmask the State: When Does State Adaptation Matter for Masked Diffusion Language Modelscited by 0
  9. 2026Simple Diffusion Language Models Are More Effective Few-Step Generators Than Reportedcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow
  2. 2025Masked Diffusion Models are Secretly Learned-Order Autoregressive Models
  3. 2025Optimizing Decoding Paths in Masked Diffusion Models by Quantifying Uncertainty
  4. 2025Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
  5. 2026Rethinking the Generation Order of Block Diffusion Language Models
  6. 2026DualDiffusion: A Speculative Decoding Strategy for Masked Diffusion Models
  7. 2025Diffusion Language Models are Provably Optimal Parallel Samplers
  8. 2026Improving Sampling for Masked Diffusion Models via Information Gain