mlsanityData to 7 Oct 2026

Structured Denoising Diffusion Models in Discrete State-Spaces

arXiv 2107.03006 ↗First posted 7 Jul 2021cs.LG cs.AI cs.CLcited by 1,665

On the map: World models → Video-generation world models and simulators

also in 1 more
Abstract (arXiv)

Denoising diffusion probabilistic models (DDPMs) (Ho et al. 2020) have shown impressive results on image and waveform generation in continuous state spaces. Here, we introduce Discrete Denoising Diffusion Probabilistic Models (D3PMs), diffusion-like generative models for discrete data that generalize the multinomial diffusion model of Hoogeboom et al. 2021, by going beyond corruption processes with uniform transition probabilities. This includes corruption with transition matrices that mimic Gaussian kernels in continuous space, matrices based on nearest neighbors in embedding space, and matrices that introduce absorbing states. The third allows us to draw a connection between diffusion models and autoregressive and mask-based generative models. We show that the choice of transition matrix is an important design decision that leads to improved results in image and text domains. We also introduce a new loss function that combines the variational lower bound with an auxiliary cross entropy loss. For text, this model class achieves strong results on character-level text generation while scaling to large vocabularies on LM1B. On the image dataset CIFAR-10, our models approach the sample quality and exceed the log-likelihood of the continuous-space DDPM model.

Key facts. Cited by 1,665 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 49 of its 71 references that are in the map. First posted on arXiv 7 Jul 2021.

Builds on

49 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  2. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2013Auto-Encoding Variational Bayescited by 3,980
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2016Wide Residual Networkscited by 3,057
35 more
  1. 2016Improved Techniques for Training GANscited by 2,938
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  3. 2015Variational Inference with Normalizing Flowscited by 2,161
  4. 2020Extracting Training Data from Large Language Modelscited by 2,144
  5. 2016Density estimation using Real NVPcited by 2,029
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  8. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2018Group Normalizationcited by 1,514
  11. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  12. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  13. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  14. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  15. 2016Pixel Recurrent Neural Networkscited by 814
  16. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  17. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  18. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  19. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  20. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  21. 2020Permutation Invariant Graph Generation via Score-Based Generative Modelingcited by 263
  22. 2019Levenshtein Transformercited by 197
  23. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  24. 2021Symbolic Music Generation with Diffusion Modelscited by 152
  25. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 151
  26. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  27. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137
  28. 2013Generalized Denoising Auto-Encoders as Generative Modelscited by 113
  29. 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
  30. 2019Discrete Flows: Invertible Generative Models of Discrete Datacited by 68
  31. 2019Latent Normalizing Flows for Discrete Sequencescited by 66
  32. 2020Imputer: Sequence Modelling via Imputation and Dynamic Programmingcited by 65
  33. 2017Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Netcited by 30
  34. 2019Discrete Object Generation with Reversible Inductive Constructioncited by 16
  35. 2020Insertion-Deletion Transformercited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 76; 2023: 202; 2024: 250; 2025: 467; 2026: 654.

1,665 papers.

Sort:

By area:

  1. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  2. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  4. 2022Diffusion Models in Vision: A Surveycited by 1,008
  5. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  6. 2025Large Language Diffusion Modelscited by 872
  7. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 835
  8. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  9. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  10. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  11. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  12. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
38 more
  1. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 666
  2. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  3. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  4. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  5. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 526
  6. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  7. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  8. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 503
  9. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  10. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  11. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 439
  12. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  13. 2023Universal Guidance for Diffusion Modelscited by 426
  14. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  15. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  16. 2022TabDDPM: Modelling Tabular Data with Diffusion Modelscited by 373
  17. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  18. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  19. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  20. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  21. 2022MAGVIT: Masked Generative Video Transformercited by 341
  22. 2024Discrete Flow Matchingcited by 301
  23. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 294
  24. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Modelscited by 279
  25. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  26. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  27. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  28. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  29. 2022A Survey on Generative Diffusion Modelcited by 238
  30. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusioncited by 230
  31. 2023Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localizationcited by 230
  32. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  33. 2023DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimizationcited by 216
  34. 2022Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptionscited by 215
  35. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  36. 2021Autoregressive Diffusion Modelscited by 208
  37. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  38. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 204

50 most cited of 1,665 All 1,665 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Discrete Diffusion Models for Language Generation
  2. 2024Discrete vs. Continuous Trade-offs for Generative Models
  3. 2024Think While You Generate: Discrete Diffusion with Planned Denoising
  4. 2023On Inference Stability for Diffusion Models
  5. 2026Discrete Diffusion Models: A Unified Framework from Tokenization to Generation
  6. 2022Image Embedding for Denoising Generative Models
  7. 2024Denoising Diffusion Probabilistic Models in Six Simple Steps
  8. 2021Bilateral Denoising Diffusion Models