mlsanityData to 7 Oct 2026

Understanding and Accelerating the Training of Masked Diffusion Language Models

arXiv 2605.13026 ↗First posted 13 May 2026cs.LG cs.AI cs.CLcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling. However, MDMs are known to learn substantially more slowly than ARMs, which may become problematic when scaling MDMs to larger models. Therefore, we ask the following question: how can we accelerate standard MDM training while maintaining its final performance? To this end, we first provide a detailed analysis of why MDM training is slow. We find that the main factor is the locality bias of language: the predictive information for a token is concentrated in nearby positions. We further investigate how this bias slows learning and suggest a simple yet effective remedy: bell-shaped time sampling as a training strategy. Notably, MDMs trained with our training recipe reach the same validation negative log-likelihood (NLL) up to ~4× faster than standard training on One Billion Word Benchmark (LM1B). We also show faster improvements in generative perplexity, zero-shot perplexity, and downstream task performance on various benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 67 of its 76 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

67 of 76 references.

Closest ancestors · 35

Same direction, newest first

  1. 2026Adaptive Guidance for Retrieval-Augmented Masked Diffusion Models18 Mar
  2. 2026Scaling Beyond Masked Diffusion Language Models16 Feb
  3. 2026Unifying Masked Diffusion Models with Various Generation Orders and Beyond2 Feb
  4. 2026Unlocking the Potentials of Retrieval-Augmented Generation for Diffusion Language Models16 Jan
  5. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100B10 Dec
  6. 2025Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models22 Nov
  7. 2025Lookahead Unmasking Elicits Accurate Decoding in Diffusion Language Models4 Nov
  8. 2025CANDI: Hybrid Discrete-Continuous Diffusion Models26 Oct
  9. 2025Soft-Masked Diffusion Language Models20 Oct
  10. 2025Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policies7 Oct
  11. 2025Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion6 Oct
  12. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner3 Oct
23 more
  1. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling1 Oct
  2. 2025Planner Aware Path Learning in Diffusion Language Models Training27 Sep
  3. 2025Dream-Coder 7B: An Open Diffusion Language Model for Code1 Sep
  4. 2025Any-Order Flexible Length Masked Diffusion31 Aug
  5. 2025Dream 7B: Diffusion Large Language Models21 Aug
  6. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference4 Aug
  7. 2025Diffusion Beats Autoregressive in Data-Constrained Settings21 Jul
  8. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  9. 2025Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture24 Jun
  10. 2025The Diffusion Duality12 Jun
  11. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  12. 2025TESS 2: A Large-Scale Generalist Diffusion Language Model19 Feb
  13. 2025Large Language Diffusion Models14 Feb
  14. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions10 Feb
  15. 2024Scaling up Masked Diffusion Models on Text24 Oct
  16. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  17. 2024Discrete Copula Diffusion2 Oct
  18. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  19. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  20. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  21. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  22. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  23. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov

Other references · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
20 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  7. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  8. 2021Variational Diffusion Modelscited by 1,145
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  11. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  12. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  13. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  14. 2022Perception Prioritized Training of Diffusion Modelscited by 301
  15. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  16. 2024Distillation of Discrete Diffusion through Dimensional Correlationscited by 46
  17. 2024Improved Noise Schedule for Diffusion Trainingcited by 38
  18. 2024Evaluating the design space of diffusion-based generative modelscited by 25
  19. 2025Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modelingcited by 14
  20. 2026Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Modelcited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimal Inference Schedules for Masked Diffusion Models
  2. 2025Theoretical Benefit and Limitation of Diffusion Language Model
  3. 2026Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison
  4. 2026Auto-Regressive Masked Diffusion Models
  5. 2026Multi-Mask Diffusion Language Models for Few-Step Generation
  6. 2026On the Trainability of Masked Diffusion Language Models via Blockwise Locality
  7. 2025Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models
  8. 2026Infinite Mask Diffusion for Few-Step Distillation