mlsanityData to 7 Oct 2026

Diffusion Beats Autoregressive in Data-Constrained Settings

arXiv 2507.15857 ↗First posted 21 Jul 2025cs.LG cs.AI cs.CVcited by 61

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Autoregressive (AR) models have long dominated the landscape of large language models, driving progress across a wide range of tasks. Recently, diffusion-based language models have emerged as a promising alternative, though their advantages over AR models remain underexplored. In this paper, we systematically study masked diffusion models in data-constrained settings where training involves repeated passes over limited data and find that they significantly outperform AR models when compute is abundant but data is scarce. Diffusion models make better use of repeated data, achieving lower validation loss and superior downstream performance. We find new scaling laws for diffusion models and derive a closed-form expression for the critical compute threshold at which diffusion begins to outperform AR. Finally, we explain why diffusion models excel in this regime: their randomized masking objective implicitly trains over a rich distribution of token orderings, acting as an implicit data augmentation that AR's fixed left-to-right factorization lacks. Our results suggest that when data, not compute, is the bottleneck, diffusion models offer a compelling alternative to the standard AR paradigm. Our code is available at: https://diffusion-scaling.github.io.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 36 of its 46 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

36 of 46 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  2. 2025Large Language Diffusion Models14 Feb
  3. 2024Scaling up Masked Diffusion Models on Text24 Oct
  4. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  5. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  6. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  7. 2023Likelihood-Based Diffusion Language Models30 May

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2023Qwen Technical Reportcited by 3,273
17 more
  1. 2016Pointer Sentinel Mixture Modelscited by 2,992
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  5. 2020GLU Variants Improve Transformercited by 1,724
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  7. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  8. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  9. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  10. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  11. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  12. 2023Scaling Data-Constrained Language Modelscited by 391
  13. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  14. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  15. 2024Randomized Autoregressive Visual Generationcited by 125
  16. 2022Image Data Augmentation for Deep Learning: A Surveycited by 96
  17. 2025Unified Multimodal Discrete Diffusioncited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 41.

61 papers.

Sort:

By area:

  1. 2025AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Sizecited by 30
  2. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimizationcited by 30
  3. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 30
  4. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  5. 2025Pre-training under infinite computecited by 24
  6. 2025From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Modelscited by 22
  7. 2025Encoder-Decoder Diffusion Language Models for Efficient Training and Inferencecited by 21
  8. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 19
  9. 2026Confidence-Based Decoding is Provably Efficient for Diffusion Language Modelscited by 17
  10. 2025Inpainting-Guided Policy Optimization for Diffusion Large Language Modelscited by 16
  11. 2025Soft-Masked Diffusion Language Modelscited by 15
  12. 2025Beyond Multi-Token Prediction: Pretraining LLMs with Future Summariescited by 14
38 more
  1. 2025DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusioncited by 13
  2. 2026Improved Large Language Diffusion Modelscited by 12
  3. 2025UALM: Unified Audio Language Model for Understanding, Generation and Reasoningcited by 10
  4. 2025Planned Diffusioncited by 8
  5. 2026Reasoning with Latent Tokens in Diffusion Language Modelscited by 8
  6. 2025RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidancecited by 7
  7. 2026Data-efficient pre-training by scaling synthetic megadocscited by 7
  8. 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
  9. 2025Masked Diffusion for Generative Recommendationcited by 5
  10. 2025Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Datacited by 5
  11. 2026Next-Scale Autoregressive Models for Text-to-Motion Generationcited by 4
  12. 2026Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minimacited by 4
  13. 2026DreamReasoner-8B: Block-Size Curriculum Learning for Diffusion Reasoning Modelscited by 4
  14. 2025Rethinking Reasoning with MDLMs: Early Exits, Post-hoc Reasoning, and Beyondcited by 3
  15. 2026MaDiS: Taming Masked Diffusion Language Models for Sign Language Generationcited by 3
  16. 2026Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretrainingcited by 3
  17. 2026Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generationcited by 3
  18. 2025DiffuTester: Accelerating Unit Test Generation for Diffusion LLMs via Mining Structural Patterncited by 2
  19. 2025Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMscited by 2
  20. 2026Plan, Verify and Fill: A Structured Parallel Decoding Approach for Diffusion Language Modelscited by 2
  21. 2026Ambient Dataloops: Generative Models for Dataset Refinementcited by 2
  22. 2026Locally Coherent Parallel Decoding in Diffusion Language Modelscited by 2
  23. 2026FastDiSS: Few-step Match Many-step Diffusion Language Model on Sequence-to-Sequence Generation--Full Versioncited by 2
  24. 2026JEDI: Joint Embedding Diffusion World Model for Online Model-Based Reinforcement Learningcited by 2
  25. 2026Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decodingcited by 2
  26. 2026DICE: Diffusion Large Language Models Excel at Generating CUDA Kernelscited by 1
  27. 2026VDLM: Variable Diffusion LMs via Robust Latent-to-Text Renderingcited by 1
  28. 2026Summer-22B: A Systematic Approach to Dataset Engineering and Training at Scale for Video Foundation Modelcited by 1
  29. 2026DiffuMask: Diffusion Language Model for Token-level Prompt Pruningcited by 1
  30. 2026The two clocks and the innovation window: When and how generative models learn rulescited by 1
  31. 2026Differences in Text Generated by Diffusion and Autoregressive Language Modelscited by 1
  32. 2026Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generatorscited by 1
  33. 2026BlockBatch: Multi-Scale Consensus Decoding for Efficient Diffusion Language Model Inferencecited by 1
  34. 2026Adaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Modelscited by 1
  35. 2026Abra: Scaling Diffusion Image Trainingcited by 1
  36. 2025Double Descent as a Lens for Sample Efficiency in Autoregressive vs. Discrete Diffusion Modelscited by 0
  37. 2026Training Language Models with homotokens Leads to Delayed Overfittingcited by 0
  38. 2026Are Video Generation Models Geographically Fair? An Attraction-Centric Evaluation of Global Visual Knowledgecited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Beyond Masked Diffusion Language Models
  2. 2025Diffusion Language Models are Super Data Learners
  3. 2026Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison
  4. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models
  5. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMs
  6. 2026Optimizers for Diffusion Models: A Controlled Benchmark
  7. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models
  8. 2025A Survey on Diffusion Language Models