# What Makes Diffusion Language Models Super Data Learners?

arXiv: https://arxiv.org/abs/2510.04071 · HTML: https://mlsanity.com/paper/2510.04071/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 5 Oct 2025.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 10 of 10)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Will we run out of data? Limits of LLM scaling based on human-generated data](https://mlsanity.com/paper/2211.04325/) (2022)
- [To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis](https://mlsanity.com/paper/2305.13230/) (2023)
- [Data Management For Training Large Language Models: A Survey](https://mlsanity.com/paper/2312.01700/) (2023)

## Built on by (most cited first, 8 of 8)

- [Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model](https://mlsanity.com/paper/2601.15892/) (2026)
- [Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation](https://mlsanity.com/paper/2606.06712/) (2026)
- [Loop the Loopies!](https://mlsanity.com/paper/2607.16051/) (2026)
- [Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data](https://mlsanity.com/paper/2512.23422/) (2025)
- [Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws](https://mlsanity.com/paper/2606.06888/) (2026)
- [Differences in Text Generated by Diffusion and Autoregressive Language Models](https://mlsanity.com/paper/2605.12522/) (2026)
- [Reinforcement Learning from Denoising Feedback](https://mlsanity.com/paper/2605.25638/) (2026)
- [Ambient Discrete Diffusion: Using the Wrong Data at the Right Time for Data Efficient Learning](https://mlsanity.com/paper/2610.12340/) (2026)

## Abstract (arXiv)

Recent studies have shown that diffusion language models achieve remarkable data efficiency under limited-data constraints, yet the underlying mechanisms remain unclear. In this work, we perform extensive ablation experiments to disentangle the sources of this efficiency. Our results show that random masking of input tokens plays the dominant role. We further show that similar gains can be obtained through in MLP dropout and weight decay, indicating that stochastic regularization broadly enhances data efficiency in multi-epoch training. Our code is available at https://github.com/zitian-gao/data-efficiency.