mlsanityData to 14 Oct 2026

Scaling Behavior of Discrete Diffusion Language Models

arXiv 2512.10858 ↗First posted 11 Dec 2025cs.LGcited by 35

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Modern LLM pre-training consumes vast amounts of compute and training data, making the scaling behavior, or scaling laws, of different models a key distinguishing factor. Discrete diffusion language models (DLMs) have been proposed as an alternative to autoregressive language models (ALMs). However, their scaling behavior has not yet been fully explored, with prior work suggesting that they require more data and compute to match the performance of ALMs. We study the scaling behavior of DLMs on different noise types by smoothly interpolating between masked and uniform diffusion while paying close attention to crucial hyperparameters such as batch size and learning rate. Our experiments reveal that the scaling behavior of DLMs strongly depends on the noise type and is considerably different from ALMs. While all noise types converge to similar loss values in compute-bound scaling, we find that uniform diffusion requires more parameters and less data for compute-efficient training compared to masked diffusion, making them a promising candidate in data-bound settings. We scale our uniform diffusion model up to 10B parameters trained for 1022 FLOPs, confirming the predicted scaling behavior and making it the largest publicly known uniform diffusion model to date.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 60 of its 65 references that are in the map. First posted on arXiv 11 Dec 2025.

Builds on

60 of 65 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Training Optimal Large Diffusion Language Models28 Sep
  2. 2025A Survey on Diffusion Language Models14 Aug
  3. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing8 Aug
  4. 2025Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models6 Jul
  5. 2025The Diffusion Duality12 Jun
  6. 2025Generalized Interpolating Discrete Diffusion6 Mar
  7. 2025Large Language Diffusion Models14 Feb
  8. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions10 Feb
  9. 2024Scaling up Masked Diffusion Models on Text24 Oct
  10. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  11. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  12. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun

Other references · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
36 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  4. 2019Root Mean Square Layer Normalizationcited by 1,480
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2021Variational Diffusion Modelscited by 1,153
  8. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  10. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  11. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  12. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  13. 2025Kimi K2: Open Agentic Intelligencecited by 567
  14. 2021Intriguing Properties of Vision Transformerscited by 446
  15. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  16. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  17. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 304
  18. 2024Massive Activations in Large Language Modelscited by 294
  19. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  20. 2018An Empirical Model of Large-Batch Trainingcited by 281
  21. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  22. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  23. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  24. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 155
  25. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 102
  26. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 82
  27. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  28. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  29. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 44
  30. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
  31. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 34
  32. 2024Scaling Law for Language Models Training Considering Batch Sizecited by 22
  33. 2020LaProp: Separating Momentum and Adaptivity in Adamcited by 20
  34. 2025Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusioncited by 15
  35. 2024Large Vocabulary Size Improves Large Language Modelscited by 11
  36. 2024Understanding and Minimising Outlier Features in Neural Network Trainingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 35.

35 papers.

Sort:

By area:

  1. 2026Categorical Flow Mapscited by 44
  2. 2026Scaling Beyond Masked Diffusion Language Modelscited by 27
  3. 2026Language Modeling with Hyperspherical Flowscited by 22
  4. 2026Efficient Sampling with Discrete Diffusion Models: Sharp and Adaptive Guaranteescited by 19
  5. 2026Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Modelcited by 13
  6. 2026Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulationcited by 13
  7. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 10
  8. 2026Locally Coherent Parallel Decoding in Diffusion Language Modelscited by 3
  9. 2026Generalized Discrete Diffusion from Snapshotscited by 3
  10. 2026Sumi: Open Uniform Diffusion Language Model from Scratchcited by 3
  11. 2026Adaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Modelscited by 2
  12. 2026Simplex Relaxation for Discrete Diffusioncited by 2
23 more
  1. 2026D5P4: Partition Determinantal Point Process for Diversity in Parallel Discrete Diffusion Decodingcited by 1
  2. 2026GeoBlock: Inferring Block Granularity from Dependency Geometry in Diffusion Language Modelscited by 1
  3. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Modelscited by 1
  4. 2026Measuring Temporal Linguistic Emergence in Diffusion Language Modelscited by 1
  5. 2026Scaling Properties of Continuous Diffusion Spoken Language Modelscited by 1
  6. 2026HIVE: Hidden-Evidence Verification for Hallucination Detection in Diffusion Large Language Modelscited by 1
  7. 2026How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Sizecited by 1
  8. 2026Training Hybrid Block Diffusion Language Models with Partial Bidirectionalitycited by 1
  9. 2026From Interface to Inference: Eliciting Any-Order Inference from Any-Order Modelscited by 1
  10. 2026LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Modelscited by 1
  11. 2026Scaling Inherently Interpretable Language Modelscited by 1
  12. 2026Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Losscited by 1
  13. 2026Stratified Hazard Sampling: Minimal-Variance Event Scheduling for CTMC/DTMC Discrete Diffusion and Flow Modelscited by 0
  14. 2026Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fitscited by 0
  15. 2026Diffusion Language Models for Speech Recognitioncited by 0
  16. 2026The Efficiency Gap in Byte Modelingcited by 0
  17. 2026Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Designcited by 0
  18. 2026Retrofitting Linear Attention into Diffusion Language Modelscited by 0
  19. 2026Optimizers for Diffusion Models: A Controlled Benchmarkcited by 0
  20. 2026Does Uniform Discrete Diffusion Need Time?cited by 0
  21. 2026Less Uniform Discrete Diffusion is More Powerful and Scalablecited by 0
  22. 2026How to scale your HEP ML models: A recipe for robust architecture comparisons at scalecited by 0
  23. 2026Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
  2. 2026Parallelism, critical windows, and separations among diffusion language models
  3. 2023Likelihood-Based Diffusion Language Models
  4. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps
  5. 2026Consistent Diffusion Language Models
  6. 2026Understanding and Accelerating the Training of Masked Diffusion Language Models
  7. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  8. 2025Diffusion Beats Autoregressive in Data-Constrained Settings