mlsanityData to 14 Oct 2026

SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers

arXiv 2212.10325 ↗First posted 20 Dec 2022cs.CLcited by 64

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion model, a new generative modelling paradigm, has achieved great success in image, audio, and video generation. However, considering the discrete categorical nature of text, it is not trivial to extend continuous diffusion models to natural language, and text diffusion models are less studied. Sequence-to-sequence text generation is one of the essential natural language processing topics. In this work, we apply diffusion models to approach sequence-to-sequence text generation, and explore whether the superiority generation performance of diffusion model can transfer to natural language domain. We propose SeqDiffuSeq, a text diffusion model for sequence-to-sequence generation. SeqDiffuSeq uses an encoder-decoder Transformers architecture to model denoising function. In order to improve generation quality, SeqDiffuSeq combines the self-conditioning technique and a newly proposed adaptive noise schedule technique. The adaptive noise schedule has the difficulty of denoising evenly distributed across time steps, and considers exclusive noise schedules for tokens at different positional order. Experiment results illustrate the good performance on sequence-to-sequence generation in terms of text quality and inference time.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 37 of its 62 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

37 of 62 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Empowering Diffusion Models on the Embedding Space for Text Generation19 Dec
  2. 2022Continuous diffusion for categorical data28 Nov
  3. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  4. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov
  5. 2022DiffusER: Discrete Diffusion via Edit-based Reconstruction30 Oct
  6. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct
  7. 2022Latent Diffusion Energy-Based Model for Interpretable Text Modeling13 Jun

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2013Auto-Encoding Variational Bayescited by 4,053
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
18 more
  1. 2022Video Diffusion Modelscited by 2,305
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  7. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  9. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  10. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  11. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
  12. 2019Levenshtein Transformercited by 199
  13. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  14. 2020Neural CRF Model for Sentence Alignment in Text Simplificationcited by 106
  15. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  16. 2018Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speechcited by 55
  17. 2017Quasar: Datasets for Question Answering by Search and Readingcited by 50
  18. 2021Non-Autoregressive Translation by Learning Target Categorical Codescited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 25; 2024: 16; 2025: 12; 2026: 11.

64 papers.

Sort:

By area:

  1. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  2. 2023Likelihood-Based Diffusion Language Modelscited by 151
  3. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generationcited by 118
  4. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  5. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 98
  6. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  7. 2025A Survey on Diffusion Language Modelscited by 78
  8. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusioncited by 65
  9. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noisescited by 50
  10. 2026ELF: Embedded Language Flowscited by 45
  11. 2023Diffusion Models for Non-autoregressive Text Generation: A Surveycited by 44
  12. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuningcited by 39
38 more
  1. 2023DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Modelscited by 36
  2. 2025DrDiff: Dynamic Routing Diffusion with Hierarchical Attention for Breaking the Efficiency-Quality Trade-offcited by 34
  3. 2025CANDI: Hybrid Discrete-Continuous Diffusion Modelscited by 34
  4. 2023DiffuSum: Generation Enhanced Extractive Summarization with Diffusioncited by 25
  5. 2025Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modelingcited by 25
  6. 2023Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generationcited by 21
  7. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generationcited by 20
  8. 2023EHRDiff: Exploring Realistic EHR Synthesis with Diffusion Modelscited by 19
  9. 2023ParaGuide: Guided Diffusion Paraphrasers for Plug-and-Play Textual Style Transfercited by 18
  10. 2023Diffusion Models in NLP: A Surveycited by 16
  11. 2023A Survey of Diffusion Models in Natural Language Processingcited by 16
  12. 2023LongDanceDiff: Long-term Dance Generation with Conditional Diffusion Modelcited by 12
  13. 2025DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Modelscited by 12
  14. 2024Diffusion on language model encodings for protein sequence generationcited by 11
  15. 2024Glauber Generative Model: Discrete Diffusion Models via Binary Classificationcited by 11
  16. 2024Text-guided multi-property molecular optimization with a diffusion language modelcited by 11
  17. 2026How to Train Your Latent Diffusion Language Model Jointly With the Latent Spacecited by 11
  18. 2026Posterior Refinement: Fast Language Generation via Any-Order Flow Mapscited by 11
  19. 2024Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decodingcited by 10
  20. 2024LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation?cited by 9
  21. 2024Discrete Diffusion Language Model for Efficient Text Summarizationcited by 8
  22. 2025Smoothie: Smoothing Diffusion on Token Embeddings for Text Generationcited by 8
  23. 2025Masked Diffusion for Generative Recommendationcited by 7
  24. 2023David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMscited by 6
  25. 2024Transfer Learning for Text Diffusion Modelscited by 6
  26. 2023DiffusionNER: Boundary Diffusion for Named Entity Recognitioncited by 5
  27. 2023InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generationcited by 5
  28. 2024Conditional [MASK] Discrete Diffusion Language Modelcited by 5
  29. 2026Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?cited by 5
  30. 2023Automatic and Human-AI Interactive Text Generationcited by 4
  31. 2024Joint Generative Modeling of Grounded Scene Graphs and Images via Diffusion Modelscited by 4
  32. 2024MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Productioncited by 4
  33. 2023Diffusion Language Models Generation Can Be Halted Earlycited by 3
  34. 2023XDLM: Cross-lingual Diffusion Language Model for Machine Translationcited by 3
  35. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  36. 2026Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Modelscited by 3
  37. 2026Forward-Learned Discrete Diffusion: Learning how to noise to denoise fastercited by 3
  38. 2023Enhancing Phrase Representation by Information Bottleneck Guided Text Diffusion Process for Keyphrase Extractioncited by 2

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Meta-DiffuB: A Contextualized Sequence-to-Sequence Text Diffusion Model with Meta-Exploration
  2. 2026One-Step Text Generation by Seq-Drifting
  3. 2025Discrete Diffusion Models for Language Generation
  4. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise
  5. 2026Diffusion Language Models: An Experimental Analysis
  6. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noise
  7. 2025State Fourier Diffusion Language Model (SFDLM): A Scalable, Novel Iterative Approach to Language Modeling
  8. 2023Can Diffusion Model Achieve Better Performance in Text Generation? Bridging the Gap between Training and Inference!