mlsanityData to 14 Oct 2026

Likelihood-Based Diffusion Language Models

arXiv 2305.18619 ↗First posted 30 May 2023cs.CL cs.LGcited by 151

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Despite a growing interest in diffusion-based language models, existing work has not shown that these models can attain nontrivial likelihoods on standard language modeling benchmarks. In this work, we take the first steps towards closing the likelihood gap between autoregressive and diffusion-based language models, with the goal of building and releasing a diffusion model which outperforms a small but widely-known autoregressive model. We pursue this goal through algorithmic improvements, scaling laws, and increased compute. On the algorithmic front, we introduce several methodological improvements for the maximum-likelihood training of diffusion language models. We then study scaling laws for our diffusion models and find compute-optimal training regimes which differ substantially from autoregressive models. Using our methods and scaling analysis, we train and release Plaid 1B, a large diffusion language model which outperforms GPT-2 124M in likelihood on benchmark datasets and generates fluent samples in unconditional and zero-shot control settings.

Key facts. Cited by 151 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 27 of its 39 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

27 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises20 Feb
  2. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers20 Dec
  3. 2022Latent Diffusion for Language Generation19 Dec
  4. 2022Continuous diffusion for categorical data28 Nov
  5. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov
  6. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  7. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2021Variational Diffusion Modelscited by 1,153
8 more
  1. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  2. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  3. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  4. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  5. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
  6. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  7. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  8. 2020Limitations of Autoregressive Models and Their Alternativescited by 55

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 23; 2025: 62; 2026: 63.

151 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 912
  2. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  3. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  4. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  5. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  6. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  7. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  8. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 305
  9. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274
  10. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  11. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 224
  12. 2024Scaling up Masked Diffusion Models on Textcited by 199
38 more
  1. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  2. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 178
  3. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  4. 2025The Diffusion Dualitycited by 135
  5. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 129
  6. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 129
  7. 2024Energy-Based Diffusion Language Models for Text Generationcited by 89
  8. 2025A Survey on Latent Reasoningcited by 81
  9. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  10. 2025Path Planning for Masked Diffusion Model Samplingcited by 80
  11. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Timecited by 78
  12. 2025A Survey on Diffusion Language Modelscited by 78
  13. 2024Dual Diffusion for Unified Image Generation and Understandingcited by 76
  14. 2024Fisher Flow Matching for Generative Modeling over Discrete Datacited by 72
  15. 2024Amortizing intractable inference in diffusion models for vision, language, and controlcited by 72
  16. 2025Generalized Interpolating Discrete Diffusioncited by 70
  17. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 63
  18. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  19. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 57
  20. 2025Unified Multimodal Discrete Diffusioncited by 49
  21. 2025Theoretical Benefit and Limitation of Diffusion Language Modelcited by 47
  22. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  23. 2026ELF: Embedded Language Flowscited by 45
  24. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuningcited by 39
  25. 2026LangFlow: Continuous Diffusion Rivals Discrete in Language Modelingcited by 36
  26. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 34
  27. 2025CANDI: Hybrid Discrete-Continuous Diffusion Modelscited by 34
  28. 2025Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Modelscited by 33
  29. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasonercited by 32
  30. 2025Debiasing Guidance for Discrete Diffusion with Sequential Monte Carlocited by 31
  31. 2025Continuous Diffusion Model for Language Modelingcited by 31
  32. 2025Self Speculative Decoding for Diffusion Large Language Modelscited by 30
  33. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 29
  34. 2026Scaling Beyond Masked Diffusion Language Modelscited by 27
  35. 2025Target Concrete Score Matching: A Holistic Framework for Discrete Diffusioncited by 25
  36. 2025Encoder-Decoder Diffusion Language Models for Efficient Training and Inferencecited by 24
  37. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Modelscited by 23
  38. 2024σ-GPTs: A New Approach to Autoregressive Modelscited by 20

50 most cited of 151 All 151 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models
  2. 2026Generative Frontiers: Why Evaluation Matters for Diffusion Language Models
  3. 2026Diffusion Language Models: An Experimental Analysis
  4. 2023David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMs
  5. 2025Diffusion Language Models are Super Data Learners
  6. 2026Sumi: Open Uniform Diffusion Language Model from Scratch
  7. 2026Enhancing Diffusion Language Models with Autoregressive Post-Training Weights
  8. 2025Scaling Behavior of Discrete Diffusion Language Models