mlsanityData to 7 Oct 2026

What Makes Diffusion Language Models Super Data Learners?

arXiv 2510.04071 ↗First posted 5 Oct 2025cs.CLcited by 8

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Recent studies have shown that diffusion language models achieve remarkable data efficiency under limited-data constraints, yet the underlying mechanisms remain unclear. In this work, we perform extensive ablation experiments to disentangle the sources of this efficiency. Our results show that random masking of input tokens plays the dominant role. We further show that similar gains can be obtained through in MLP dropout and weight decay, indicating that stochastic regularization broadly enhances data efficiency in multi-epoch training. Our code is available at https://github.com/zitian-gao/data-efficiency.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 10 of its 18 references that are in the map. First posted on arXiv 5 Oct 2025.

Builds on

10 of 18 references.

References · 10

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  8. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  9. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 88
  10. 2023Data Management For Training Large Language Models: A Surveycited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2026Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Modelcited by 12
  2. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  3. 2025Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Datacited by 5
  4. 2026Loop the Loopies!cited by 4
  5. 2026Differences in Text Generated by Diffusion and Autoregressive Language Modelscited by 1
  6. 2026Reinforcement Learning from Denoising Feedbackcited by 0
  7. 2026Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Lawscited by 0
  8. 2026Ambient Discrete Diffusion: Using the Wrong Data at the Right Time for Data Efficient Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion Beats Autoregressive in Data-Constrained Settings
  2. 2025Diffusion Language Models are Super Data Learners
  3. 2026Understanding and Accelerating the Training of Masked Diffusion Language Models
  4. 2025Theoretical Benefit and Limitation of Diffusion Language Model
  5. 2026Scaling Beyond Masked Diffusion Language Models
  6. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
  7. 2025Simple Denoising Diffusion Language Models
  8. 2026Continuous Diffusion Models Can Obey Formal Syntax