mlsanityData to 7 Oct 2026

Diffusion Models for Non-autoregressive Text Generation: A Survey

arXiv 2303.06574 ↗First posted 12 Mar 2023cs.CLcited by 43

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Non-autoregressive (NAR) text generation has attracted much attention in the field of natural language processing, which greatly reduces the inference latency but has to sacrifice the generation accuracy. Recently, diffusion models, a class of latent variable generative models, have been introduced into NAR text generation, showing an improved text generation quality. In this survey, we review the recent progress in diffusion models for NAR text generation. As the background, we first present the general definition of diffusion models and the text diffusion models, and then discuss their merits for NAR generation. As the core content, we further introduce two mainstream diffusion models in existing work of text diffusion, and review the key designs of the diffusion process. Moreover, we discuss the utilization of pre-trained language models (PLMs) for text diffusion models and introduce optimization techniques for text data. Finally, we discuss several promising directions and conclude this paper. Our survey aims to provide researchers with a systematic reference of related research on text diffusion models for NAR generation. We present our collection of text diffusion models at https://github.com/RUCAIBox/Awesome-Text-Diffusion-Models.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 44 of its 63 references that are in the map. First posted on arXiv 12 Mar 2023.

Builds on

44 of 63 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises20 Feb
  2. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  3. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers20 Dec
  4. 2022Latent Diffusion for Language Generation19 Dec
  5. 2022Continuous diffusion for categorical data28 Nov
  6. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  7. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov
  8. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  9. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
23 more
  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  6. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  9. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  10. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  11. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  12. 2017Non-Autoregressive Neural Machine Translationcited by 457
  13. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  14. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  15. 2016SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documentscited by 245
  16. 2017Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluationcited by 206
  17. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  18. 2019Levenshtein Transformercited by 197
  19. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 121
  20. 2021Pretrained Language Models for Text Generation: A Surveycited by 95
  21. 2022Unified Discrete Diffusion for Simultaneous Vision-Language Generationcited by 39
  22. 2022Semantic-Conditional Diffusion Networks for Image Captioningcited by 38
  23. 2020BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretrainingcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 12; 2025: 17; 2026: 7.

43 papers.

Sort:

By area:

  1. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  2. 2024An analytic theory of creativity in convolutional diffusion modelscited by 103
  3. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  4. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  5. 2023Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimizationcited by 56
  6. 2023Diffusion Augmentation for Sequential Recommendationcited by 37
  7. 2023Foundation Models for Weather and Climate Data Understanding: A Comprehensive Surveycited by 32
  8. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  9. 2023Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generationcited by 21
  10. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generationcited by 19
  11. 2025Empirical Analysis of Decoding Biases in Masked Diffusion Modelscited by 19
  12. 2025ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decodingcited by 16
31 more
  1. 2025Deep Learning and Foundation Models for Weather Prediction: A Surveycited by 15
  2. 2025Unveiling the Potential of Diffusion Large Language Model in Controllable Generationcited by 14
  3. 2025Implicit Search via Discrete Diffusion: A Study on Chesscited by 11
  4. 2024Generative AI for Controllable Protein Sequence Design: A Surveycited by 8
  5. 2024Conditional score-based diffusion models for solving inverse problems in mechanicscited by 6
  6. 2024Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Surveycited by 6
  7. 2023EVE: Efficient zero-shot text-based Video Editing with Depth Map Guidance and Temporal Consistency Constraintscited by 5
  8. 2023InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generationcited by 5
  9. 2025Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Modelscited by 5
  10. 2024Segment-Level Diffusion: A Framework for Controllable Long-Form Generation with Diffusion Language Modelscited by 4
  11. 2025Visual Watermarking in the Era of Diffusion Models: Advances and Challengescited by 4
  12. 2025WavefrontDiffusion: Dynamic Decoding Schedule for Improved Reasoningcited by 4
  13. 2024Text Diffusion with Reinforced Conditioningcited by 3
  14. 2024MaskPure: Improving Defense Against Text Adversaries with Stochastic Purificationcited by 2
  15. 2024Data Augmentation via Diffusion Model to Enhance AI Fairnesscited by 2
  16. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Developmentcited by 2
  17. 2025Don't Let It Fade: Preserving Edits in Diffusion Language Models via Token Timestep Allocationcited by 2
  18. 2024Reinforcement Learning for Edit-Based Non-Autoregressive Neural Machine Translationcited by 1
  19. 2024Erase, then Redraw: A Novel Data Augmentation Approach for Free Space Detection Using Diffusion Modelcited by 1
  20. 2025Non-asymptotic convergence bound of conditional diffusion modelscited by 1
  21. 2025Alternatives To Next Token Prediction In Text Generation -- A Surveycited by 1
  22. 2026Can I Have Your Order? Monte-Carlo Tree Search for Slot Filling Ordering in Diffusion Language Modelscited by 1
  23. 2025Weak Supervision Dynamic KL-Weighted Diffusion Models Guided by Large Language Modelscited by 0
  24. 2025Discriminative protein sequence modelling with Latent Space Diffusioncited by 0
  25. 2025Syntax-Guided Diffusion Language Models with User-Integrated Personalizationcited by 0
  26. 2026Zonkey: A Hierarchical Diffusion Language Model with Differentiable Tokenization and Probabilistic Attentioncited by 0
  27. 2026Generative AI and Federated Learning for Intrusion Detection Systems: A Surveycited by 0
  28. 2026Diffusion Language Model for Recommendationcited by 0
  29. 2026Information Spreading in Diffusion Models from Effective Field Theorycited by 0
  30. 2026Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoderscited by 0
  31. 2026Eval4DiRec: A Unified and Systematic Evaluation Framework for Diffusion-based Recommender Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey of Diffusion Models in Natural Language Processing
  2. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
  3. 2023Diffusion Models in NLP: A Survey
  4. 2024Energy-Based Diffusion Language Models for Text Generation
  5. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model
  6. 2026Towards Latent Diffusion Suitable For Text
  7. 2023Can Diffusion Model Achieve Better Performance in Text Generation? Bridging the Gap between Training and Inference!
  8. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusion