mlsanityData to 7 Oct 2026

Self-conditioned Embedding Diffusion for Text Generation

arXiv 2211.04236 ↗First posted 8 Nov 2022cs.CL cs.LGcited by 114

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Can continuous diffusion models bring the same performance breakthrough on natural language they did for image generation? To circumvent the discrete nature of text data, we can simply project tokens in a continuous space of embeddings, as is standard in language modeling. We propose Self-conditioned Embedding Diffusion, a continuous diffusion mechanism that operates on token embeddings and allows to learn flexible and scalable diffusion models for both conditional and unconditional text generation. Through qualitative and quantitative evaluation, we show that our text diffusion models generate samples comparable with those produced by standard autoregressive language models - while being in theory more efficient on accelerator hardware at inference time. Our work paves the way for scaling up diffusion models for text, similarly to autoregressive models, and for improving performance with recent refinements to continuous diffusion.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 30 of its 35 references that are in the map. First posted on arXiv 8 Nov 2022.

Builds on

30 of 35 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
18 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  8. 2021Variational Diffusion Modelscited by 1,145
  9. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  10. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  11. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  12. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  13. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  14. 2021Symbolic Music Generation with Diffusion Modelscited by 152
  15. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  16. 2018Language GANs Falling Shortcited by 97
  17. 2019Training language GANs from Scratchcited by 31
  18. 2018On Accurate Evaluation of GANs for Language Generationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 31; 2024: 23; 2025: 24; 2026: 30.

114 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 872
  2. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  3. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  4. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  5. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 294
  6. 2022Continuous diffusion for categorical datacited by 202
  7. 2024Scaling up Masked Diffusion Models on Textcited by 195
  8. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  9. 2023Protein Design with Guided Discrete Diffusioncited by 154
  10. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  11. 2023Likelihood-Based Diffusion Language Modelscited by 147
  12. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
38 more
  1. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 124
  2. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 118
  3. 2022Latent Diffusion for Language Generationcited by 110
  4. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  5. 2023DiffuRec: A Diffusion Model for Sequential Recommendationcited by 91
  6. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoisecited by 79
  7. 2025A Survey on Diffusion Language Modelscited by 75
  8. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformerscited by 63
  9. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusioncited by 63
  10. 2023Bayesian Flow Networkscited by 62
  11. 2024Discrete Copula Diffusioncited by 53
  12. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noisescited by 50
  13. 2023Diffusion Models for Non-autoregressive Text Generation: A Surveycited by 43
  14. 2023FinDiff: Diffusion Models for Financial Tabular Data Generationcited by 41
  15. 2022Empowering Diffusion Models on the Embedding Space for Text Generationcited by 40
  16. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Modelscited by 32
  17. 2025CANDI: Hybrid Discrete-Continuous Diffusion Modelscited by 31
  18. 2026ELF: Embedded Language Flowscited by 27
  19. 2023DiffuSum: Generation Enhanced Extractive Summarization with Diffusioncited by 25
  20. 2023PLay: Parametrically Conditioned Layout Generation using Latent Diffusioncited by 24
  21. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Modelscited by 23
  22. 2024TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodingscited by 22
  23. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generationcited by 19
  24. 2023ParaGuide: Guided Diffusion Paraphrasers for Plug-and-Play Textual Style Transfercited by 18
  25. 2025Attention Sinks in Diffusion Language Modelscited by 18
  26. 2023Diffusion Models in NLP: A Surveycited by 16
  27. 2023A Survey of Diffusion Models in Natural Language Processingcited by 16
  28. 2023Continuous Diffusion for Mixed-Type Tabular Datacited by 16
  29. 2025Soft-Masked Diffusion Language Modelscited by 15
  30. 2025Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Modelscited by 13
  31. 2026CoDAR: Continuous Diffusion Language Models are More Powerful Than You Thinkcited by 13
  32. 2026Continuous Diffusion Scales Competitively with Discrete Diffusion for Languagecited by 13
  33. 2025LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understandingcited by 12
  34. 2026Continuous Latent Diffusion Language Modelcited by 12
  35. 2024Diffusion on language model encodings for protein sequence generationcited by 11
  36. 2024Glauber Generative Model: Discrete Diffusion Models via Binary Classificationcited by 11
  37. 2024Promises, Outlooks and Challenges of Diffusion Language Modelingcited by 11
  38. 2024Text-guided multi-property molecular optimization with a diffusion language modelcited by 11

50 most cited of 114 All 114 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
  2. 2026Scaling and Distilling Text Embeddings for Better Diffusibility
  3. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models
  4. 2026AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling
  5. 2023Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation
  6. 2025Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes
  7. 2025Smoothie: Smoothing Diffusion on Token Embeddings for Text Generation
  8. 2025Discrete Diffusion Models for Language Generation