mlsanityData to 14 Oct 2026

Diffusion Language Models are Super Data Learners

arXiv 2511.03276 ↗First posted 5 Nov 2025cs.LGcited by 48

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Under strictly controlled pre-training settings, we observe a Crossover: when unique data is limited, diffusion language models (DLMs) consistently surpass autoregressive (AR) models by training for more epochs. The crossover shifts later with more or higher-quality data, earlier with larger models, and persists across dense and sparse architectures. We attribute the gains to three compounding factors: (1) any-order modeling, (2) super-dense compute from iterative bidirectional denoising, and (3) built-in Monte Carlo augmentation; input or parameter noise improves AR under data constraint but cannot close the gap. At scale, a 1.7B DLM trained with a ~1.5T-token compute budget on 10B unique Python tokens overtakes an AR coder trained with strictly matched settings. In addition, a 1B-parameter DLM achieves > 56% accuracy on HellaSwag and > 33% on MMLU using only 1B tokens, without any special tricks, just by repeating standard pre-training data. We also show that rising validation cross-entropy does not imply degraded downstream performance in this regime.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 38 of its 45 references that are in the map. First posted on arXiv 5 Nov 2025.

Builds on

38 of 45 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Dream-Coder 7B: An Open Diffusion Language Model for Code1 Sep
  2. 2025Dream 7B: Diffusion Large Language Models21 Aug
  3. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference4 Aug
  4. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  5. 2025Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture24 Jun
  6. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  7. 2025Large Language Diffusion Models14 Feb
  8. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  9. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  10. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  11. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  9. 2025Kimi K2: Open Agentic Intelligencecited by 567
  10. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  11. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  12. 2023Scaling Data-Constrained Language Modelscited by 403
15 more
  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  2. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  3. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  4. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  5. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  6. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  7. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 130
  8. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 92
  9. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 84
  10. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 56
  11. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 52
  12. 2025DataDecide: How to Predict Best Pretraining Data with Small Experimentscited by 40
  13. 2025Rewriting Pre-Training Data Boosts LLM Performance in Math and Codecited by 21
  14. 2025Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusioncited by 15
  15. 2024Rephrasing natural text data with different languages and quality levels for Large Language Model pre-trainingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 43.

48 papers.

Sort:

By area:

  1. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 34
  2. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  3. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 18
  4. 2026Improved Large Language Diffusion Modelscited by 14
  5. 2026Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Modelcited by 13
  6. 2026Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusioncited by 13
  7. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  8. 2026Where-to-Unmask: Ground-Truth-Guided Unmasking Order Learning for Masked Diffusion Language Modelscited by 9
  9. 2026Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuningcited by 8
  10. 2026MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decodingcited by 8
  11. 2026When to Commit? Towards Variable-Size Self-Contained Blocks for Discrete Diffusion Language Modelscited by 7
  12. 2025DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbonecited by 6
36 more
  1. 2026FLARE: Diffusion for Hybrid Language Modelcited by 6
  2. 2025Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Datacited by 5
  3. 2026DreamReasoner-8B: Block-Size Curriculum Learning for Diffusion Reasoning Modelscited by 5
  4. 2025On the Role of Discreteness in Diffusion LLMscited by 4
  5. 2026Training-Trajectory-Aware Token Selectioncited by 4
  6. 2026Next-Scale Autoregressive Models for Text-to-Motion Generationcited by 4
  7. 2026Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minimacited by 4
  8. 2026Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretrainingcited by 4
  9. 2026Top 10 Open Challenges Steering the Future of Diffusion Language Model and Its Variantscited by 3
  10. 2026Causal Autoregressive Diffusion Language Modelcited by 3
  11. 2026DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understandingcited by 3
  12. 2026FOCUS: DLLMs Know How to Tame Their Compute Boundcited by 3
  13. 2026DICE: Diffusion Large Language Models Excel at Generating CUDA Kernelscited by 3
  14. 2026Locally Coherent Parallel Decoding in Diffusion Language Modelscited by 3
  15. 2026Fast Byte Latent Transformercited by 3
  16. 2026Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Lawscited by 3
  17. 2026Sink-Aware Pruning for Diffusion Language Modelscited by 2
  18. 2026On the Trainability of Masked Diffusion Language Models via Blockwise Localitycited by 2
  19. 2026Diffusion Language Models: An Experimental Analysiscited by 2
  20. 2026Adaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Modelscited by 2
  21. 2026Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scalecited by 2
  22. 2026Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from k-Paritycited by 1
  23. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Modelscited by 1
  24. 2026The two clocks and the innovation window: When and how generative models learn rulescited by 1
  25. 2026Differences in Text Generated by Diffusion and Autoregressive Language Modelscited by 1
  26. 2026LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Modelscited by 1
  27. 2026Diffusion Language Models for Speech Recognitioncited by 0
  28. 2026BadDLM: Backdooring Diffusion Language Models with Diverse Targetscited by 0
  29. 2026Reinforcement Learning from Denoising Feedbackcited by 0
  30. 2026Destruction is a General Strategy to Learn Generation; Diffusion's Strength is to Take it Seriously; Exploration is the Futurecited by 0
  31. 2026Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Designcited by 0
  32. 2026Diffusion Language Model for Recommendationcited by 0
  33. 2026Affix Cache for Diffusion Large Language Modelscited by 0
  34. 2026Ambient Discrete Diffusion: Using the Wrong Data at the Right Time for Data Efficient Learningcited by 0
  35. 2026Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Modelscited by 0
  36. 2026Diff Diffusion: Predicting Edit Paths with Diffusion Models for Efficient Code Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion Beats Autoregressive in Data-Constrained Settings
  2. 2025A Survey on Diffusion Language Models
  3. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMs
  4. 2025TESS 2: A Large-Scale Generalist Diffusion Language Model
  5. 2026Enhancing Diffusion Language Models with Autoregressive Post-Training Weights
  6. 2023Likelihood-Based Diffusion Language Models
  7. 2026Introspective Diffusion Language Models
  8. 2026Optimizers for Diffusion Models: A Controlled Benchmark