mlsanityData to 7 Oct 2026

Scaling up Masked Diffusion Models on Text

arXiv 2410.18514 ↗First posted 24 Oct 2024cs.AI cs.CL cs.LGcited by 195

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Masked diffusion models (MDMs) have shown promise in language modeling, yet their scalability and effectiveness in core language tasks, such as text generation and language understanding, remain underexplored. This paper establishes the first scaling law for MDMs, demonstrating a scaling rate comparable to autoregressive models (ARMs) and a relatively small compute gap. Motivated by their scalability, we train a family of MDMs with up to 1.1 billion (B) parameters to systematically evaluate their performance against ARMs of comparable or larger sizes. Fully leveraging the probabilistic formulation of MDMs, we propose a simple yet effective unsupervised classifier-free guidance that effectively exploits large-scale unpaired data, boosting performance for conditional inference. In language understanding, the 1.1B MDM outperforms the 1.1B TinyLlama model trained on the same data across four of eight zero-shot benchmarks. Notably, it achieves competitive math reasoning ability with the 7B Llama-2 model on the GSM8K dataset. In text generation, MDMs with 16 times more pre-training time offer a flexible trade-off against ARMs with the accelerated sampling technique KV-Cache: MDMs match ARMs in performance while being 1.4 times faster during sampling. Moreover, MDMs address challenging tasks for ARMs by effectively handling bidirectional reasoning and adapting to temporal shifts in data. Notably, a 1.1B MDM breaks the reverse curse encountered by much larger ARMs with significantly more data and computation, such as 13B Llama-2 and 175B GPT-3. Our code is available at https://github.com/ML-GSAI/SMDM.

Key facts. Cited by 195 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 66 of its 81 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

66 of 81 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  2. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  3. 2024Diffusion Guided Language Modeling8 Aug
  4. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  5. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  6. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  7. 2024Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models12 Feb
  8. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning23 Aug
  9. 2023Likelihood-Based Diffusion Language Models30 May
  10. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusion15 May
  11. 2022Continuous diffusion for categorical data28 Nov
  12. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
4 more
  1. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov
  2. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  3. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct
  4. 2021Autoregressive Diffusion Models5 Oct

Other references · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  11. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  12. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
38 more
  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  13. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  14. 2020GLU Variants Improve Transformercited by 1,724
  15. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  16. 2019Root Mean Square Layer Normalizationcited by 1,425
  17. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  18. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  19. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  20. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  21. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  22. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  23. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  24. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  25. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  26. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  27. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  28. 2024Discrete Flow Matchingcited by 301
  29. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  30. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 167
  31. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 165
  32. 2022Concrete Score Matching: Generalized Score Matching for Discrete Datacited by 144
  33. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  34. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  35. 2024CLLMs: Consistency Large Language Modelscited by 71
  36. 2023Bayesian Flow Networkscited by 62
  37. 2024Unifying Bayesian Flow Networks and Diffusion Models through Stochastic Differential Equationscited by 28
  38. 2024The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and Morecited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 96; 2026: 96.

195 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 872
  2. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  3. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  4. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  5. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  6. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  7. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  8. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  9. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  10. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 128
  11. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 118
  12. 2025Path Planning for Masked Diffusion Model Samplingcited by 79
38 more
  1. 2025A Survey on Latent Reasoningcited by 71
  2. 2025Generalized Interpolating Discrete Diffusioncited by 68
  3. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  4. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  5. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 61
  6. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
  7. 2025Any-Order Flexible Length Masked Diffusioncited by 48
  8. 2025Theoretical Benefit and Limitation of Diffusion Language Modelcited by 46
  9. 2025Unified Multimodal Discrete Diffusioncited by 46
  10. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  11. 2025Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principlescited by 43
  12. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  13. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 38
  14. 2025DPad: Efficient Diffusion Language Models with Suffix Dropoutcited by 38
  15. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 38
  16. 2025Plan for Speed: Dilated Scheduling for Masked Diffusion Language Modelscited by 34
  17. 2025Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Modelscited by 33
  18. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 32
  19. 2025Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Modelscited by 32
  20. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 32
  21. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  22. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 32
  23. 2025CANDI: Hybrid Discrete-Continuous Diffusion Modelscited by 31
  24. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 30
  25. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 30
  26. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  27. 2025Jailbreaking Large Language Diffusion Models: Revealing Hidden Safety Flaws in Diffusion-Based Text Generationcited by 24
  28. 2025d2Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Cachingcited by 24
  29. 2026Scaling Beyond Masked Diffusion Language Modelscited by 22
  30. 2025Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Maskingcited by 21
  31. 2025Training Optimal Large Diffusion Language Modelscited by 21
  32. 2025Accelerating Diffusion LLM Inference via Local Determinism Propagationcited by 21
  33. 2025Encoder-Decoder Diffusion Language Models for Efficient Training and Inferencecited by 21
  34. 2025Absorb and Converge: Provable Convergence Guarantee for Absorbing Discrete Diffusion Modelscited by 20
  35. 2025CDLM: Consistency Diffusion Language Models For Faster Samplingcited by 20
  36. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generationcited by 19
  37. 2025Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spacescited by 19
  38. 2025Empirical Analysis of Decoding Biases in Masked Diffusion Modelscited by 19

50 most cited of 195 All 195 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Representation-based Masked Diffusion Model
  2. 2025On the Reasoning Abilities of Masked Diffusion Language Models
  3. 2025Optimal Inference Schedules for Masked Diffusion Models
  4. 2026Infinite Mask Diffusion for Few-Step Distillation
  5. 2026Diffusion Language Models: An Experimental Analysis
  6. 2026Unifying Masked Diffusion Models with Various Generation Orders and Beyond
  7. 2026On the Trainability of Masked Diffusion Language Models via Blockwise Locality
  8. 2026CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language