# Scaling Beyond Masked Diffusion Language Models

arXiv: https://arxiv.org/abs/2602.15014 · HTML: https://mlsanity.com/paper/2602.15014/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 37 of its 44 references that are in the map.
- First posted on arXiv 16 Feb 2026.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data](https://mlsanity.com/paper/2406.03736/) (2024)
- [From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step](https://mlsanity.com/paper/2405.14838/) (2024)
- [Mercury: Ultra-Fast Language Models Based on Diffusion](https://mlsanity.com/paper/2506.17298/) (2025)
- [Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling](https://mlsanity.com/paper/2409.02908/) (2024)
- [Scaling up Masked Diffusion Models on Text](https://mlsanity.com/paper/2410.18514/) (2024)
- [Remasking Discrete Diffusion Models with Inference-Time Scaling](https://mlsanity.com/paper/2503.00307/) (2025)
- [Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference](https://mlsanity.com/paper/2508.02193/) (2025)
- [Simple Guidance Mechanisms for Discrete Diffusion Models](https://mlsanity.com/paper/2412.10193/) (2024)

## Built on by (most cited first, 27 of 27)

- [ELF: Embedded Language Flows](https://mlsanity.com/paper/2605.10938/) (2026)
- [Continuous Diffusion Scales Competitively with Discrete Diffusion for Language](https://mlsanity.com/paper/2605.18530/) (2026)
- [DMax: Aggressive Parallel Decoding for dLLMs](https://mlsanity.com/paper/2604.08302/) (2026)
- [The Diffusion Duality, Chapter II: Psi-Samplers](https://mlsanity.com/paper/2602.21185/) (2026)
- [Scaling Categorical Flow Maps](https://mlsanity.com/paper/2605.07820/) (2026)
- [The Design Space of Tri-Modal Masked Diffusion Models](https://mlsanity.com/paper/2602.21472/) (2026)
- [FLARE: Diffusion for Hybrid Language Model](https://mlsanity.com/paper/2606.01774/) (2026)
- [DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling](https://mlsanity.com/paper/2605.23605/) (2026)
- [Sumi: Open Uniform Diffusion Language Model from Scratch](https://mlsanity.com/paper/2606.19005/) (2026)
- [Drifting Objectives for Refining Discrete Diffusion Language Models](https://mlsanity.com/paper/2605.19470/) (2026)
- [Recursive Scaling in Masked Diffusion Models](https://mlsanity.com/paper/2606.18022/) (2026)
- [Simplex Relaxation for Discrete Diffusion](https://mlsanity.com/paper/2608.10615/) (2026)
- [ConvergeFlow: Language Flow with Provable Convergence to Token Embeddings](https://mlsanity.com/paper/2608.23551/) (2026)
- [Unlocking Lossless Speedups in LLMs via Discrete Diffusion](https://mlsanity.com/paper/2609.04010/) (2026)
- [What Does a Discrete Diffusion Model Learn?](https://mlsanity.com/paper/2607.05381/) (2026)
- [LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models](https://mlsanity.com/paper/2608.03457/) (2026)
- [Scaling Inherently Interpretable Language Models](https://mlsanity.com/paper/2608.07594/) (2026)
- [Parallelism, critical windows, and separations among diffusion language models](https://mlsanity.com/paper/2609.20539/) (2026)
- [Diffusion Language Models for Speech Recognition](https://mlsanity.com/paper/2604.14001/) (2026)
- [Understanding and Accelerating the Training of Masked Diffusion Language Models](https://mlsanity.com/paper/2605.13026/) (2026)
- [Uncertainty Quantification for Large Language Diffusion Models](https://mlsanity.com/paper/2605.14570/) (2026)
- [Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design](https://mlsanity.com/paper/2606.15327/) (2026)
- [How to Guide Your Language Flow](https://mlsanity.com/paper/2609.19356/) (2026)
- [Does Uniform Discrete Diffusion Need Time?](https://mlsanity.com/paper/2609.30977/) (2026)
- [On the Design Space of Continuous Diffusion Language Models](https://mlsanity.com/paper/2610.13834/) (2026)
- [Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models](https://mlsanity.com/paper/2610.17590/) (2026)
- [The Tokens Strike Back: Safety-Aware Denoising with Directional Embedding-based Revision](https://mlsanity.com/paper/2610.17636/) (2026)

## Abstract (arXiv)

Diffusion language models are a promising alternative to autoregressive models due to their potential for faster generation. Among discrete diffusion approaches, Masked diffusion currently dominates, largely driven by strong perplexity on language modeling benchmarks. In this work, we present the first scaling law study of uniform-state and interpolating discrete diffusion methods. We also show that Masked diffusion models can be made approximately 12% more FLOPs-efficient when trained with a simple cross-entropy objective. We find that perplexity is informative within a diffusion family but can be misleading across families, where models with worse likelihood scaling may be preferable due to faster and more practical sampling, as reflected by the speed-quality Pareto frontier. These results challenge the view that Masked diffusion is categorically the future of diffusion language modeling and that perplexity alone suffices for cross-algorithm comparison. Scaling all methods to 1.7B parameters, we show that uniform-state diffusion remains competitive on likelihood-based benchmarks and outperforms autoregressive and Masked diffusion models on GSM8K, despite worse validation perplexity. We provide the code, model checkpoints, and video tutorials on the project page: http://s-sahoo.github.io/scaling-dllms