mlsanityData to 7 Oct 2026

Step-unrolled Denoising Autoencoders for Text Generation

arXiv 2112.06749 ↗First posted 13 Dec 2021cs.CL cs.LGcited by 102

Not on the map. Why?

Cited by 67 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper we propose a new generative model of text, Step-unrolled Denoising Autoencoder (SUNDAE), that does not rely on autoregressive models. Similarly to denoising diffusion techniques, SUNDAE is repeatedly applied on a sequence of tokens, starting from random inputs and improving them each time until convergence. We present a simple new improvement operator that converges in fewer iterations than diffusion methods, while qualitatively producing better samples on natural language datasets. SUNDAE achieves state-of-the-art results (among non-autoregressive methods) on the WMT'14 English-to-German translation task and good qualitative results on unconditional language modeling on the Colossal Cleaned Common Crawl dataset and a dataset of Python code from GitHub. The non-autoregressive nature of SUNDAE opens up possibilities beyond left-to-right prompted generation, by filling in arbitrary blank patterns in a template.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Builds on 51 of its 70 references that are in the map. First posted on arXiv 13 Dec 2021.

Builds on

51 of 70 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  12. 2013Auto-Encoding Variational Bayescited by 3,980
38 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2015Variational Inference with Normalizing Flowscited by 2,161
  4. 2016Density estimation using Real NVPcited by 2,029
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  8. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  9. 2016Sequence-Level Knowledge Distillationcited by 761
  10. 2015Generating Sentences from a Continuous Spacecited by 607
  11. 2017Non-Autoregressive Neural Machine Translationcited by 457
  12. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  13. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  14. 2019Levenshtein Transformercited by 197
  15. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  16. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  17. 2021Symbolic Music Generation with Diffusion Modelscited by 152
  18. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 123
  19. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 103
  20. 2017Long Text Generation via Adversarial Training with Leaked Informationcited by 100
  21. 2018Language GANs Falling Shortcited by 97
  22. 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
  23. 2020Residual Energy-Based Models for Text Generationcited by 95
  24. 2019FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flowcited by 89
  25. 2019Preventing Posterior Collapse with delta-VAEscited by 86
  26. 2020Imputer: Sequence Modelling via Imputation and Dynamic Programmingcited by 65
  27. 2019Fast Structured Decoding for Sequence Modelscited by 59
  28. 2019Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posteriorcited by 58
  29. 2019Non-Autoregressive Machine Translation with Auxiliary Regularizationcited by 52
  30. 2020Pre-Training Transformers as Energy-Based Cloze Modelscited by 52
  31. 2017Adversarial Ranking for Language Generationcited by 50
  32. 2017Maximum-Likelihood Augmented Discrete Generative Adversarial Networkscited by 39
  33. 2019Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Informationcited by 37
  34. 2020Semi-Autoregressive Training Improves Mask-Predict Decodingcited by 33
  35. 2019Training language GANs from Scratchcited by 31
  36. 2019Hint-Based Training for Non-Autoregressive Machine Translationcited by 28
  37. 2020Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translationcited by 13
  38. 2020Incorporating a Local Translation Mechanism into Non-autoregressive Translationcited by 9

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 36; 2024: 21; 2025: 11; 2026: 14.

102 papers.

Sort:

By area:

  1. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  2. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  3. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 439
  4. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  5. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  6. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  7. 2024Discrete Flow Matchingcited by 301
  8. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  9. 2022Continuous diffusion for categorical datacited by 202
  10. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  11. 2023Protein Design with Guided Discrete Diffusioncited by 154
  12. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
38 more
  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  2. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  3. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  4. 2023The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimationcited by 116
  5. 2022Self-conditioned Embedding Diffusion for Text Generationcited by 114
  6. 2023Monocular Depth Estimation using Diffusion Modelscited by 87
  7. 2024Energy-Based Diffusion Language Models for Text Generationcited by 86
  8. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  9. 2023Diffusion Models for Time Series Applications: A Surveycited by 72
  10. 2023Structure-informed Language Models Are Protein Designerscited by 68
  11. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  12. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformerscited by 63
  13. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusioncited by 63
  14. 2024Informed Correctors for Discrete Diffusion Modelscited by 57
  15. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noisescited by 50
  16. 2025Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithmscited by 47
  17. 2022Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiationcited by 46
  18. 2022CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Languagecited by 46
  19. 2022Categorical SDEs with Simplex Diffusioncited by 42
  20. 2022DiffusER: Discrete Diffusion via Edit-based Reconstructioncited by 41
  21. 2022Empowering Diffusion Models on the Embedding Space for Text Generationcited by 40
  22. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuningcited by 39
  23. 2023An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systemscited by 33
  24. 2024Attacks and Defenses for Generative Diffusion Models: A Comprehensive Surveycited by 32
  25. 2022Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformercited by 30
  26. 2025TESS 2: A Large-Scale Generalist Diffusion Language Modelcited by 30
  27. 2025Target Concrete Score Matching: A Holistic Framework for Discrete Diffusioncited by 25
  28. 2023Conditional Denoising Diffusion for Sequential Recommendationcited by 23
  29. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noisecited by 21
  30. 2023Quantum-Noise-Driven Generative Diffusion Modelscited by 20
  31. 2023Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Timecited by 20
  32. 2022Lossless Acceleration for Seq2seq Generation with Aggressive Decodingcited by 18
  33. 2024Arrows of Time for Large Language Modelscited by 18
  34. 2024Collaborative Filtering Based on Diffusion Models: Unveiling the Potential of High-Order Connectivitycited by 18
  35. 2023Diffusion Models in NLP: A Surveycited by 16
  36. 2023A Survey of Diffusion Models in Natural Language Processingcited by 16
  37. 2025Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorizationcited by 16
  38. 2023Diffusion Models in Bioinformatics: A New Wave of Deep Learning Revolution in Actioncited by 15

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise
  2. 2022ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generation
  3. 2024A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models
  4. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
  5. 2023Diffusion Models for Non-autoregressive Text Generation: A Survey
  6. 2025Discrete Diffusion Models for Language Generation
  7. 2025Simple Denoising Diffusion Language Models
  8. 2026Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Models