mlsanityData to 14 Oct 2026

Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions

arXiv 2502.06768 ↗First posted 10 Feb 2025cs.LGcited by 214

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

In recent years, masked diffusion models (MDMs) have emerged as a promising alternative approach for generative modeling over discrete domains. Compared to autoregressive models (ARMs), MDMs trade off complexity at training time with flexibility at inference time. At training time, they must learn to solve an exponentially large number of infilling problems, but at inference time, they can decode tokens in essentially arbitrary order. In this work, we closely examine these two competing effects. On the training front, we theoretically and empirically demonstrate that MDMs indeed train on computationally intractable subproblems compared to their autoregressive counterparts. On the inference front, we show that a suitable strategy for adaptively choosing the token decoding order significantly enhances the capabilities of MDMs, allowing them to sidestep hard subproblems. On logic puzzles like Sudoku, we show that adaptive inference can boost solving accuracy in pretrained MDMs from <7% to ≈ 90%, even outperforming ARMs with 7× as many parameters and that were explicitly trained via teacher forcing to learn the right order of decoding.

Key facts. Cited by 214 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 45 of its 58 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

45 of 58 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Path Planning for Masked Diffusion Model Sampling5 Feb
  2. 2024Energy-Based Diffusion Language Models for Text Generation28 Oct
  3. 2024Scaling up Masked Diffusion Models on Text24 Oct
  4. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  5. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning18 Oct
  6. 2024Think While You Generate: Discrete Diffusion with Planned Denoising8 Oct
  7. 2024Discrete Copula Diffusion2 Oct
  8. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  9. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  10. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  11. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  12. 2024Glauber Generative Model: Discrete Diffusion Models via Binary Classification27 May
3 more
  1. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  2. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  3. 2021Autoregressive Diffusion Models5 Oct

Other references · 30

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  9. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  10. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  11. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  12. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
18 more
  1. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  2. 2015The Benefit of Multitask Representation Learningcited by 195
  3. 2020Provable Meta-Learning of Linear Representationscited by 161
  4. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 155
  5. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 98
  6. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 90
  7. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 72
  8. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  9. 2024Reverse Training to Nurse the Reversal Cursecited by 49
  10. 2024Convergence Analysis of Discrete Diffusion Model: Exact Implementation through Uniformizationcited by 44
  11. 2024Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzlescited by 38
  12. 2024Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Predictioncited by 32
  13. 2020Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Ordercited by 28
  14. 2024The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and Morecited by 25
  15. 2024Arrows of Time for Large Language Modelscited by 20
  16. 2024Hardness of sampling solutions from the Symmetric Binary Perceptroncited by 8
  17. 2024Task Diversity Shortens the ICL Plateaucited by 5
  18. 2021On statistical inference when fixed points of belief propagation are unstablecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 78; 2026: 136.

214 papers.

Sort:

By area:

  1. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  2. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 143
  3. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 131
  4. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 112
  5. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  6. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  7. 2025Generalized Interpolating Discrete Diffusioncited by 70
  8. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  9. 2025Any-Order Flexible Length Masked Diffusioncited by 51
  10. 2025Theoretical Benefit and Limitation of Diffusion Language Modelcited by 47
  11. 2025KLASS: KL-Guided Fast Inference in Masked Diffusion Modelscited by 47
  12. 2025ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMscited by 43
38 more
  1. 2025Fine-Tuning Masked Diffusion for Provable Self-Correctioncited by 39
  2. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 35
  3. 2025Plan for Speed: Dilated Scheduling for Masked Diffusion Language Modelscited by 34
  4. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 34
  5. 2025Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Modelscited by 33
  6. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimizationcited by 33
  7. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Modelscited by 32
  8. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasonercited by 32
  9. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  10. 2025Learning Unmasking Policies for Diffusion Language Modelscited by 30
  11. 2025From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMscited by 24
  12. 2025From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Modelscited by 23
  13. 2025Sudoku-Bench: Evaluating creative reasoning with Sudoku variantscited by 22
  14. 2025Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policiescited by 22
  15. 2025MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Controlcited by 21
  16. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 20
  17. 2025Empirical Analysis of Decoding Biases in Masked Diffusion Modelscited by 19
  18. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learningcited by 19
  19. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  20. 2025ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decodingcited by 18
  21. 2026Confidence-Based Decoding is Provably Efficient for Diffusion Language Modelscited by 18
  22. 2025Unifying Autoregressive and Diffusion-Based Sequence Generationcited by 17
  23. 2025Lookahead Unmasking Elicits Accurate Decoding in Diffusion Language Modelscited by 17
  24. 2025Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecturecited by 16
  25. 2025Ultra-Fast Language Generation via Discrete Diffusion Divergence Instructcited by 16
  26. 2026OAT: Ordered Action Tokenizationcited by 15
  27. 2025Planner Aware Path Learning in Diffusion Language Models Trainingcited by 14
  28. 2025Error Bounds and Optimal Schedules for Masked Diffusions with Factorized Approximationscited by 14
  29. 2026DAPD: Dependency-Aware Parallel Decoding via Attention for Diffusion LLMscited by 14
  30. 2025CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Creditcited by 13
  31. 2025Parallel Sampling from Masked Diffusion Models via Conditional Independence Testingcited by 13
  32. 2026Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Modelcited by 13
  33. 2026Learn from Your Mistakes: Self-Correcting Masked Diffusion Modelscited by 13
  34. 2025Insertion Language Models: Sequence Generation with Arbitrary-Position Insertionscited by 11
  35. 2025The Cosine Schedule is Fisher-Rao-Optimal for Masked Discrete Diffusion Modelscited by 11
  36. 2026Breaking the Factorization Barrier in Diffusion Language Modelscited by 11
  37. 2026Posterior Refinement: Fast Language Generation via Any-Order Flow Mapscited by 11
  38. 2025Simple and Critical Iterative Denoising: A Recasting of Discrete Diffusion in Graph Generationcited by 10

50 most cited of 214 All 214 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking
  2. 2025Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models
  3. 2026A Theoretical Analysis of Why Masked Diffusion Models Mitigate the Reversal Curse
  4. 2026On the Trainability of Masked Diffusion Language Models via Blockwise Locality
  5. 2026MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Scaling of Diffusion Language Models
  6. 2025On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond
  7. 2026Auto-Regressive Masked Diffusion Models
  8. 2025Optimal Inference Schedules for Masked Diffusion Models