mlsanityData to 7 Oct 2026

Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation

arXiv 2305.04044 ↗First posted 6 May 2023cs.CLcited by 21

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Recently, continuous diffusion models (CDM) have been introduced into non-autoregressive (NAR) text-to-text generation. However, the discrete nature of text increases the difficulty of CDM to generate coherent and fluent texts, and also causes the incompatibility problem between CDM and advanced NLP techniques, especially the popular pre-trained language models~(PLMs). To solve it, we propose Diffusion-NAT, which introduces discrete diffusion models~(DDM) into NAR text-to-text generation and integrates BART to improve the performance. By revising the decoding process of BART and the typical settings of DDM, we unify the inference process of BART and the denoising process of DDM into the same NAR masked tokens recovering task. In this way, DDM can rely on BART to perform denoising, which can benefit from both the rich pre-learned knowledge of BART and the iterative refining paradigm of DDM. Besides, we also propose the iterative self-prompting strategy to further improve the generation quality. Experimental results on 7 datasets show that our approach can outperform competitive NAR methods, and even surpass autoregressive methods. Our code and data will be publicly released.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 40 of its 57 references that are in the map. First posted on arXiv 6 May 2023.

Builds on

40 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Diffusion Models for Non-autoregressive Text Generation: A Survey12 Mar
  2. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  3. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers20 Dec
  4. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
24 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  3. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  5. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  6. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  7. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  8. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  9. 2018CoQA: A Conversational Question Answering Challengecited by 674
  10. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  11. 2017Non-Autoregressive Neural Machine Translationcited by 457
  12. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  13. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  14. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  15. 2019Levenshtein Transformercited by 197
  16. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  17. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 123
  18. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 121
  19. 2020Glancing Transformer for Non-Autoregressive Neural Machine Translationcited by 103
  20. 2021Pretrained Language Models for Text Generation: A Surveycited by 95
  21. 2020GLGE: A New General Language Generation Evaluation Benchmarkcited by 47
  22. 2020BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretrainingcited by 33
  23. 2023WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Modelscited by 23
  24. 2022ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 12; 2026: 3.

21 papers.

Sort:

By area:

  1. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  2. 2025A Survey on Diffusion Language Modelscited by 75
  3. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  4. 2024How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Frameworkcited by 48
  5. 2025Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithmscited by 47
  6. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 40
  7. 2023DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Modelscited by 36
  8. 2025Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decodingcited by 26
  9. 2025Absorb and Converge: Provable Convergence Guarantee for Absorbing Discrete Diffusion Modelscited by 20
  10. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generationcited by 19
  11. 2025Unifying Autoregressive and Diffusion-Based Sequence Generationcited by 17
  12. 2023David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMscited by 6
9 more
  1. 2023InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generationcited by 5
  2. 2024Conditional [MASK] Discrete Diffusion Language Modelcited by 4
  3. 2024Segment-Level Diffusion: A Framework for Controllable Long-Form Generation with Diffusion Language Modelscited by 4
  4. 2025Discrete State Diffusion Models: A Sample Complexity Perspectivecited by 4
  5. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  6. 2025Bringing Emerging Architectures to Sequence Labeling in NLPcited by 1
  7. 2026Coupling Models for One-Step Discrete Generationcited by 1
  8. 2026From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizonscited by 1
  9. 2026When to Plan, When to Polish: Noise Level as a Granularity Axis for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Self-conditioned Embedding Diffusion for Text Generation
  2. 2025Discrete Diffusion Models for Language Generation
  3. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
  4. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise
  5. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusion
  6. 2025Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes
  7. 2023A Survey of Diffusion Models in Natural Language Processing
  8. 2023Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Time