# Scaling Behavior of Discrete Diffusion Language Models

arXiv: https://arxiv.org/abs/2512.10858 · HTML: https://mlsanity.com/paper/2512.10858/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 60 of its 65 references that are in the map.
- First posted on arXiv 11 Dec 2025.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 30 of 60)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)

## Built on by (most cited first, 30 of 35)

- [Categorical Flow Maps](https://mlsanity.com/paper/2602.12233/) (2026)
- [Scaling Beyond Masked Diffusion Language Models](https://mlsanity.com/paper/2602.15014/) (2026)
- [Language Modeling with Hyperspherical Flows](https://mlsanity.com/paper/2605.11125/) (2026)
- [Efficient Sampling with Discrete Diffusion Models: Sharp and Adaptive Guarantees](https://mlsanity.com/paper/2602.15008/) (2026)
- [Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model](https://mlsanity.com/paper/2601.15892/) (2026)
- [Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation](https://mlsanity.com/paper/2605.22765/) (2026)
- [Deriving Hyperparameter Scaling Laws via Modern Optimization Theory](https://mlsanity.com/paper/2603.15958/) (2026)
- [Locally Coherent Parallel Decoding in Diffusion Language Models](https://mlsanity.com/paper/2603.20216/) (2026)
- [Generalized Discrete Diffusion from Snapshots](https://mlsanity.com/paper/2603.21342/) (2026)
- [Sumi: Open Uniform Diffusion Language Model from Scratch](https://mlsanity.com/paper/2606.19005/) (2026)
- [Adaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Models](https://mlsanity.com/paper/2606.29275/) (2026)
- [Simplex Relaxation for Discrete Diffusion](https://mlsanity.com/paper/2608.10615/) (2026)
- [D5P4: Partition Determinantal Point Process for Diversity in Parallel Discrete Diffusion Decoding](https://mlsanity.com/paper/2603.19146/) (2026)
- [GeoBlock: Inferring Block Granularity from Dependency Geometry in Diffusion Language Models](https://mlsanity.com/paper/2603.26675/) (2026)
- [Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models](https://mlsanity.com/paper/2604.02340/) (2026)
- [Measuring Temporal Linguistic Emergence in Diffusion Language Models](https://mlsanity.com/paper/2604.23235/) (2026)
- [Scaling Properties of Continuous Diffusion Spoken Language Models](https://mlsanity.com/paper/2604.24416/) (2026)
- [HIVE: Hidden-Evidence Verification for Hallucination Detection in Diffusion Large Language Models](https://mlsanity.com/paper/2604.26139/) (2026)
- [How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size](https://mlsanity.com/paper/2607.01487/) (2026)
- [Training Hybrid Block Diffusion Language Models with Partial Bidirectionality](https://mlsanity.com/paper/2607.02805/) (2026)
- [From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models](https://mlsanity.com/paper/2607.26504/) (2026)
- [LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models](https://mlsanity.com/paper/2608.03457/) (2026)
- [Scaling Inherently Interpretable Language Models](https://mlsanity.com/paper/2608.07594/) (2026)
- [Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss](https://mlsanity.com/paper/2608.28308/) (2026)
- [Stratified Hazard Sampling: Minimal-Variance Event Scheduling for CTMC/DTMC Discrete Diffusion and Flow Models](https://mlsanity.com/paper/2601.02799/) (2026)
- [Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits](https://mlsanity.com/paper/2603.22339/) (2026)
- [Diffusion Language Models for Speech Recognition](https://mlsanity.com/paper/2604.14001/) (2026)
- [The Efficiency Gap in Byte Modeling](https://mlsanity.com/paper/2605.12928/) (2026)
- [Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design](https://mlsanity.com/paper/2606.15327/) (2026)
- [Retrofitting Linear Attention into Diffusion Language Models](https://mlsanity.com/paper/2608.06628/) (2026)

## Abstract (arXiv)

Modern LLM pre-training consumes vast amounts of compute and training data, making the scaling behavior, or scaling laws, of different models a key distinguishing factor. Discrete diffusion language models (DLMs) have been proposed as an alternative to autoregressive language models (ALMs). However, their scaling behavior has not yet been fully explored, with prior work suggesting that they require more data and compute to match the performance of ALMs. We study the scaling behavior of DLMs on different noise types by smoothly interpolating between masked and uniform diffusion while paying close attention to crucial hyperparameters such as batch size and learning rate. Our experiments reveal that the scaling behavior of DLMs strongly depends on the noise type and is considerably different from ALMs. While all noise types converge to similar loss values in compute-bound scaling, we find that uniform diffusion requires more parameters and less data for compute-efficient training compared to masked diffusion, making them a promising candidate in data-bound settings. We scale our uniform diffusion model up to 10B parameters trained for 1022 FLOPs, confirming the predicted scaling behavior and making it the largest publicly known uniform diffusion model to date.