mlsanityData to 7 Oct 2026

Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise

arXiv 2212.11685 ↗First posted 22 Dec 2022cs.CL cs.LGcited by 79

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

In this paper, we introduce a novel dIffusion language modEl pre-training framework for text generation, which we call GENIE. GENIE is a large-scale pretrained diffusion language model that consists of an encoder and a diffusion-based decoder, which can generate text by gradually transforming a random noise sequence into a coherent text sequence. To pre-train GENIE on a large-scale language corpus, we design a new continuous paragraph denoise objective, which encourages the diffusion-decoder to reconstruct a clean text paragraph from a corrupted version, while preserving the semantic and syntactic coherence. We evaluate GENIE on four downstream text generation benchmarks, namely XSum, CNN/DailyMail, Gigaword, and CommonGen. Our experimental results show that GENIE achieves comparable performance with the state-of-the-art autoregressive models on these benchmarks, and generates more diverse text samples. The code and models of GENIE are available at https://github.com/microsoft/ProphetNet/tree/master/GENIE.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 38 of its 47 references that are in the map. First posted on arXiv 22 Dec 2022.

Builds on

38 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov
  2. 2022DiffusER: Discrete Diffusion via Edit-based Reconstruction30 Oct
  3. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
23 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2022Video Diffusion Modelscited by 2,280
  3. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  4. 2018Hierarchical Neural Story Generationcited by 1,182
  5. 2015Teaching Machines to Read and Comprehendcited by 1,039
  6. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  7. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 666
  8. 2017Non-Autoregressive Neural Machine Translationcited by 457
  9. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  10. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  11. 2015LSTM: A Search Space Odysseycited by 315
  12. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  13. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  14. 2016Incorporating Copying Mechanism in Sequence-to-Sequence Learningcited by 217
  15. 2019Levenshtein Transformercited by 197
  16. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  17. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 123
  18. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  19. 2018A Graph-to-Sequence Model for AMR-to-Text Generationcited by 56
  20. 2020Lexically Constrained Neural Machine Translation with Levenshtein Transformercited by 45
  21. 2021Non-Autoregressive Text Generation with Pre-trained Language Modelscited by 36
  22. 2020BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretrainingcited by 33
  23. 2022ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 23; 2025: 23; 2026: 20.

79 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 872
  2. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  3. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  4. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  5. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  6. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generationcited by 113
  7. 2025A Survey on Diffusion Language Modelscited by 75
  8. 2023A Text-guided Protein Design Frameworkcited by 61
  9. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  10. 2023DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Modelscited by 36
  11. 2025DrDiff: Dynamic Routing Diffusion with Hierarchical Attention for Breaking the Efficiency-Quality Trade-offcited by 33
  12. 2024Attacks and Defenses for Generative Diffusion Models: A Comprehensive Surveycited by 32
38 more
  1. 2023CodeFusion: A Pre-trained Diffusion Model for Code Generationcited by 31
  2. 2025Self Speculative Decoding for Diffusion Large Language Modelscited by 29
  3. 2026ELF: Embedded Language Flowscited by 27
  4. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  5. 2025Encoder-Decoder Diffusion Language Models for Efficient Training and Inferencecited by 21
  6. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generationcited by 19
  7. 2023Diffusion Models in NLP: A Surveycited by 16
  8. 2023A Survey of Diffusion Models in Natural Language Processingcited by 16
  9. 2023DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLMcited by 16
  10. 2025FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Modelscited by 16
  11. 2024Atlas Gaussians Diffusion for 3D Generationcited by 15
  12. 2026dLLM: Simple Diffusion Language Modelingcited by 12
  13. 2024Diffusion on language model encodings for protein sequence generationcited by 11
  14. 2023Instruction Position Matters in Sequence Generation with Large Language Modelscited by 10
  15. 2024Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decodingcited by 10
  16. 2025Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerabilitycited by 10
  17. 2024DiffLM: Controllable Synthetic Data Generation via Diffusion Language Modelscited by 9
  18. 2025Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transformcited by 9
  19. 2023A Wireless AI-Generated Content (AIGC) Provisioning Framework Empowered by Semantic Communicationcited by 8
  20. 2024DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreakcited by 8
  21. 2025Smoothie: Smoothing Diffusion on Token Embeddings for Text Generationcited by 8
  22. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  23. 2026Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decodingcited by 6
  24. 2026Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspectivecited by 6
  25. 2024Transfer Learning for Text Diffusion Modelscited by 5
  26. 2026Membership Inference Attacks Against Fine-tuned Diffusion Language Modelscited by 5
  27. 2024Joint Generative Modeling of Grounded Scene Graphs and Images via Diffusion Modelscited by 4
  28. 2024KS-LLM: Knowledge Selection of Large Language Models with Evidence Document for Question Answeringcited by 4
  29. 2024Latent Diffusion Models for Controllable RNA Sequence Generationcited by 4
  30. 2024Conditional [MASK] Discrete Diffusion Language Modelcited by 4
  31. 2024Segment-Level Diffusion: A Framework for Controllable Long-Form Generation with Diffusion Language Modelscited by 4
  32. 2025Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraintscited by 4
  33. 2023XDLM: Cross-lingual Diffusion Language Model for Machine Translationcited by 3
  34. 2023Adversarial Denoising Diffusion Model for Unsupervised Anomaly Detectioncited by 3
  35. 2025Diffusion is a code repair operator and generatorcited by 3
  36. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  37. 2026Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?cited by 3
  38. 2026TextLDM: Language Modeling with Continuous Latent Diffusioncited by 3

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Discrete Diffusion Models for Language Generation
  2. 2023Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation
  3. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers
  4. 2023Diffusion Models for Non-autoregressive Text Generation: A Survey
  5. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model
  6. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noise
  7. 2025Simple Denoising Diffusion Language Models
  8. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models