mlsanityData to 7 Oct 2026

Training Optimal Large Diffusion Language Models

arXiv 2510.03280 ↗First posted 28 Sep 2025cs.LG cs.AI cs.CLcited by 21

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

We introduce Quokka, the first systematic scaling law for diffusion language models (DLMs), encompassing both compute-constrained and data-constrained regimes, and studying the key modeling and optimization designs. Quokka is a good friend of Chinchilla and provides wider scopes. We hope the results would bring short-term practical guidance in DLMs training and long-term inspirations for the whole AI community.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 32 of its 43 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

32 of 43 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Dream-Coder 7B: An Open Diffusion Language Model for Code1 Sep
  2. 2025Dream 7B: Diffusion Large Language Models21 Aug
  3. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference4 Aug
  4. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  5. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  6. 2025Large Language Diffusion Models14 Feb
  7. 2024Scaling up Masked Diffusion Models on Text24 Oct
  8. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  9. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  10. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  11. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  9. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  10. 2023Scaling Data-Constrained Language Modelscited by 391
  11. 2021Explaining Neural Scaling Lawscited by 353
  12. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
9 more
  1. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  2. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  3. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  4. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  5. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  6. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 88
  7. 2024Scaling Laws for Downstream Task Performance of Large Language Modelscited by 49
  8. 2025Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusioncited by 14
  9. 2025SkyLadder: Better and Faster Pretraining via Context Window Schedulingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 12.

21 papers.

Sort:

By area:

  1. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  2. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 32
  3. 2025Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Modelscited by 19
  4. 2026Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Modelcited by 12
  5. 2025Corrective Diffusion Language Modelscited by 11
  6. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Modelscited by 10
  7. 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
  8. 2025DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbonecited by 4
  9. 2025On the Role of Discreteness in Diffusion LLMscited by 4
  10. 2026Revealing the Attention Floating Mechanism in Masked Diffusion Modelscited by 4
  11. 2025Simple Denoising Diffusion Language Modelscited by 2
  12. 2026LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Modelscited by 2
9 more
  1. 2026Locally Coherent Parallel Decoding in Diffusion Language Modelscited by 2
  2. 2026Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from k-Paritycited by 1
  3. 2026DICE: Diffusion Large Language Models Excel at Generating CUDA Kernelscited by 1
  4. 2026Training Hybrid Block Diffusion Language Models with Partial Bidirectionalitycited by 1
  5. 2026Multi-Mask Diffusion Language Models for Few-Step Generationcited by 1
  6. 2026LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Modelscited by 1
  7. 2025Latent-Augmented Discrete Diffusion Modelscited by 0
  8. 2026FourierSampler: Unlocking Non-Autoregressive Potential in Diffusion Language Models via Frequency-Guided Generationcited by 0
  9. 2026MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Scaling of Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Likelihood-Based Diffusion Language Models
  2. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models
  3. 2026Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
  4. 2026dLLM: Simple Diffusion Language Modeling
  5. 2025A Survey on Diffusion Language Models
  6. 2026Representation-Space MMD for Diffusion Language Models
  7. 2026Scaling Beyond Masked Diffusion Language Models
  8. 2026Diffusion Language Models: An Experimental Analysis