mlsanityData to 7 Oct 2026

A Survey of Diffusion Models in Natural Language Processing

arXiv 2305.14671 ↗First posted 24 May 2023cs.CLcited by 16

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

This survey paper provides a comprehensive review of the use of diffusion models in natural language processing (NLP). Diffusion models are a class of mathematical models that aim to capture the diffusion of information or signals across a network or manifold. In NLP, diffusion models have been used in a variety of applications, such as natural language generation, sentiment analysis, topic modeling, and machine translation. This paper discusses the different formulations of diffusion models used in NLP, their strengths and limitations, and their applications. We also perform a thorough comparison between diffusion models and alternative generative models, specifically highlighting the autoregressive (AR) models, while also examining how diverse architectures incorporate the Transformer in conjunction with diffusion models. Compared to AR models, diffusion models have significant advantages for parallel generation, text interpolation, token-level controls such as syntactic structures and semantic contents, and robustness. Exploring further permutations of integrating Transformers into diffusion models would be a valuable pursuit. Also, the development of multimodal diffusion models and large-scale diffusion language models with notable capabilities for few-shot learning would be important directions for the future advance of diffusion models in NLP.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 35 of its 43 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

35 of 43 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noise10 Apr
  2. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises20 Feb
  3. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  4. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise22 Dec
  5. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers20 Dec
  6. 2022Latent Diffusion for Language Generation19 Dec
  7. 2022Empowering Diffusion Models on the Embedding Space for Text Generation19 Dec
  8. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  9. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov
  10. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  11. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct
  12. 2021Zero-Shot Translation using Diffusion Models2 Nov

Other references · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
11 more
  1. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  2. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  3. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  4. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  5. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  6. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  7. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  8. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  9. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 94
  10. 2020A Systematic Characterization of Sampling Algorithms for Open-ended Language Generationcited by 28
  11. 2023DiffuSum: Generation Enhanced Extractive Summarization with Diffusioncited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 7; 2026: 4.

16 papers.

Sort:

By area:

  1. 2023FinDiff: Diffusion Models for Financial Tabular Data Generationcited by 41
  2. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  3. 2024Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectivescited by 27
  4. 2025Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedulescited by 10
  5. 2025Diffusion Models for Future Networks and Communications: A Comprehensive Surveycited by 6
  6. 2023ForceGen: End-to-end de novo protein generation based on nonlinear mechanical unfolding responses using a protein language diffusion modelcited by 5
  7. 2025Diffuse Thinking: Exploring Diffusion Language Models as Efficient Thought Proposers for Reasoningcited by 3
  8. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Developmentcited by 2
  9. 2024Cognitively Inspired Energy-Based World Modelscited by 1
  10. 2025The Effect of Stochasticity in Score-Based Diffusion Sampling: a KL Divergence Analysiscited by 1
  11. 2025Alternatives To Next Token Prediction In Text Generation -- A Surveycited by 1
  12. 2026Generalizing Score-based generative models for Heavy-tailed Distributionscited by 1
4 more
  1. 2025Effective Series Decomposition and Components Learning for Time Series Generationcited by 0
  2. 2026From Denoising to Decision Making: A Survey on Diffusion Model-Enabled Deep Reinforcement Learning for Wireless Networkscited by 0
  3. 2026Unified Energy for Invariant and Independent Decoding in Diffusion Language Modelscited by 0
  4. 2026LESS Is More: Mutual-Stability Sampling for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Diffusion Models in NLP: A Survey
  2. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models
  3. 2023Diffusion Models for Non-autoregressive Text Generation: A Survey
  4. 2025A Survey on Diffusion Language Models
  5. 2025Discrete Diffusion Models for Language Generation
  6. 2026Diffusion Language Models: An Experimental Analysis
  7. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
  8. 2026Towards Latent Diffusion Suitable For Text