# A Survey of Diffusion Models in Natural Language Processing arXiv: https://arxiv.org/abs/2305.14671 ยท HTML: https://mlsanity.com/paper/2305.14671/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. - Builds on 35 of its 43 references that are in the map. - First posted on arXiv 24 May 2023. ## Placed in - Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/) ## Builds on (most cited first, 30 of 35) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020) - [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021) - [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022) - [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021) - [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021) - [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018) - [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022) - [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017) - [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021) - [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020) - [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021) - [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022) - [Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning](https://mlsanity.com/paper/2208.04202/) (2022) - [DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models](https://mlsanity.com/paper/2211.15029/) (2022) - [SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control](https://mlsanity.com/paper/2210.17432/) (2022) - [A Reparameterized Discrete Diffusion Model for Text Generation](https://mlsanity.com/paper/2302.05737/) (2023) - [Self-conditioned Embedding Diffusion for Text Generation](https://mlsanity.com/paper/2211.04236/) (2022) - [Latent Diffusion for Language Generation](https://mlsanity.com/paper/2212.09462/) (2022) - [Step-unrolled Denoising Autoencoders for Text Generation](https://mlsanity.com/paper/2112.06749/) (2021) - [Trading Off Diversity and Quality in Natural Language Generation](https://mlsanity.com/paper/2004.10450/) (2020) - [Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise](https://mlsanity.com/paper/2212.11685/) (2022) - [SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers](https://mlsanity.com/paper/2212.10325/) (2022) - [DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises](https://mlsanity.com/paper/2302.10025/) (2023) ## Built on by (most cited first, 16 of 16) - [FinDiff: Diffusion Models for Financial Tabular Data Generation](https://mlsanity.com/paper/2309.01472/) (2023) - [Undesirable Memorization in Large Language Models: A Survey](https://mlsanity.com/paper/2410.02650/) (2024) - [Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectives](https://mlsanity.com/paper/2404.08926/) (2024) - [Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules](https://mlsanity.com/paper/2512.02892/) (2025) - [Diffusion Models for Future Networks and Communications: A Comprehensive Survey](https://mlsanity.com/paper/2508.01586/) (2025) - [ForceGen: End-to-end de novo protein generation based on nonlinear mechanical unfolding responses using a protein language diffusion model](https://mlsanity.com/paper/2310.10605/) (2023) - [Diffuse Thinking: Exploring Diffusion Language Models as Efficient Thought Proposers for Reasoning](https://mlsanity.com/paper/2510.27469/) (2025) - [Generalizing Score-based generative models for Heavy-tailed Distributions](https://mlsanity.com/paper/2603.00772/) (2026) - [Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development](https://mlsanity.com/paper/2510.12253/) (2025) - [Cognitively Inspired Energy-Based World Models](https://mlsanity.com/paper/2406.08862/) (2024) - [The Effect of Stochasticity in Score-Based Diffusion Sampling: a KL Divergence Analysis](https://mlsanity.com/paper/2506.11378/) (2025) - [Alternatives To Next Token Prediction In Text Generation -- A Survey](https://mlsanity.com/paper/2509.24435/) (2025) - [Effective Series Decomposition and Components Learning for Time Series Generation](https://mlsanity.com/paper/2511.00747/) (2025) - [From Denoising to Decision Making: A Survey on Diffusion Model-Enabled Deep Reinforcement Learning for Wireless Networks](https://mlsanity.com/paper/2605.25531/) (2026) - [Unified Energy for Invariant and Independent Decoding in Diffusion Language Models](https://mlsanity.com/paper/2606.09159/) (2026) - [LESS Is More: Mutual-Stability Sampling for Diffusion Language Models](https://mlsanity.com/paper/2606.16908/) (2026) ## Abstract (arXiv) This survey paper provides a comprehensive review of the use of diffusion models in natural language processing (NLP). Diffusion models are a class of mathematical models that aim to capture the diffusion of information or signals across a network or manifold. In NLP, diffusion models have been used in a variety of applications, such as natural language generation, sentiment analysis, topic modeling, and machine translation. This paper discusses the different formulations of diffusion models used in NLP, their strengths and limitations, and their applications. We also perform a thorough comparison between diffusion models and alternative generative models, specifically highlighting the autoregressive (AR) models, while also examining how diverse architectures incorporate the Transformer in conjunction with diffusion models. Compared to AR models, diffusion models have significant advantages for parallel generation, text interpolation, token-level controls such as syntactic structures and semantic contents, and robustness. Exploring further permutations of integrating Transformers into diffusion models would be a valuable pursuit. Also, the development of multimodal diffusion models and large-scale diffusion language models with notable capabilities for few-shot learning would be important directions for the future advance of diffusion models in NLP.