mlsanityData to 7 Oct 2026

Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution

arXiv 2310.16834 ↗First posted 25 Oct 2023stat.ML cs.CL cs.LGcited by 712

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Despite their groundbreaking performance for many generative modeling tasks, diffusion models have fallen short on discrete data domains such as natural language. Crucially, standard diffusion models rely on the well-established theory of score matching, but efforts to generalize this to discrete structures have not yielded the same empirical gains. In this work, we bridge this gap by proposing score entropy, a novel loss that naturally extends score matching to discrete spaces, integrates seamlessly to build discrete diffusion models, and significantly boosts performance. Experimentally, we test our Score Entropy Discrete Diffusion models (SEDD) on standard language modeling tasks. For comparable model sizes, SEDD beats existing language diffusion paradigms (reducing perplexity by 25-75%) and is competitive with autoregressive models, in particular outperforming GPT-2. Furthermore, compared to autoregressive mdoels, SEDD generates faithful text without requiring distribution annealing techniques like temperature scaling (around 6-8× better generative perplexity than un-annealed GPT-2), can trade compute and quality (similar quality with 32× fewer network evaluations), and enables controllable infilling (matching nucleus sampling quality while enabling other strategies besides left to right prompting).

Key facts. Cited by 712 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 40 of its 58 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

40 of 58 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Likelihood-Based Diffusion Language Models30 May
  2. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusion15 May
  3. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises20 Feb
  4. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  5. 2022Continuous diffusion for categorical data28 Nov
  6. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  7. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov
  8. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  9. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
19 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  3. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  4. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  5. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  6. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  7. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  8. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  9. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 381
  10. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  11. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  12. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 167
  13. 2022Concrete Score Matching: Generalized Score Matching for Discrete Datacited by 144
  14. 2023Reflected Diffusion Modelscited by 81
  15. 2019Discrete Flows: Invertible Generative Models of Discrete Datacited by 68
  16. 2019Latent Normalizing Flows for Discrete Sequencescited by 66
  17. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  18. 2023Bayesian Flow Networkscited by 62
  19. 2019Neural Networks with Cheap Differential Operatorscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 36; 2025: 250; 2026: 426.

712 papers.

Sort:

By area:

  1. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  2. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  3. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 294
  4. 2024Diffusion Policy Policy Optimizationcited by 255
  5. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  6. 2025Remasking Discrete Diffusion Models with Inference-Time Scalingcited by 181
  7. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  8. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  9. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  10. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  11. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 128
  12. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 118
38 more
  1. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcingcited by 111
  2. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  3. 2025How to build a consistency model: Learning flow maps via self-distillationcited by 96
  4. 2024Energy-Based Diffusion Language Models for Text Generationcited by 86
  5. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  6. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
  7. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
  8. 2025Path Planning for Masked Diffusion Model Samplingcited by 79
  9. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  10. 2025A Survey on Diffusion Language Modelscited by 75
  11. 2025Generalized Interpolating Discrete Diffusioncited by 68
  12. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  13. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  14. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 61
  15. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  16. 2024Discrete Copula Diffusioncited by 53
  17. 2025Edit Flows: Flow Matching with Edit Operationscited by 53
  18. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 52
  19. 2024How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Frameworkcited by 48
  20. 2025Any-Order Flexible Length Masked Diffusioncited by 48
  21. 2025Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithmscited by 47
  22. 2024Distillation of Discrete Diffusion through Dimensional Correlationscited by 46
  23. 2025Theoretical Benefit and Limitation of Diffusion Language Modelcited by 46
  24. 2025Unified Multimodal Discrete Diffusioncited by 46
  25. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  26. 2024OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flowscited by 45
  27. 2024ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformercited by 45
  28. 2025Diffusion Language Models are Super Data Learnerscited by 45
  29. 2025KLASS: KL-Guided Fast Inference in Masked Diffusion Modelscited by 44
  30. 2024Linear Convergence of Diffusion Models Under the Manifold Hypothesiscited by 42
  31. 2025ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMscited by 42
  32. 2025Fine-Tuning Masked Diffusion for Provable Self-Correctioncited by 39
  33. 2025DPad: Efficient Diffusion Language Models with Suffix Dropoutcited by 38
  34. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 38
  35. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  36. 2026Categorical Flow Mapscited by 37
  37. 2024PepTune: De Novo Generation of Therapeutic Peptides with Multi-Objective-Guided Discrete Diffusioncited by 35
  38. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34

50 most cited of 712 All 712 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Promises, Outlooks and Challenges of Diffusion Language Modeling
  2. 2025Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models
  3. 2024Improving Discrete Diffusion Models via Structured Preferential Generation
  4. 2025The Diffusion Duality
  5. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models
  6. 2025Understanding the Quality-Diversity Trade-off in Diffusion Language Models
  7. 2024Simple and Effective Masked Diffusion Language Models
  8. 2022How Much is Enough? A Study on Diffusion Times in Score-based Generative Models