mlsanityData to 7 Oct 2026

TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings

arXiv 2402.19097 ↗First posted 29 Feb 2024cs.CLcited by 22

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

This paper presents the Text Encoding Diffusion Model (TEncDM), a novel approach to diffusion modeling that operates in the space of pre-trained language model encodings. In contrast to traditionally used embeddings, encodings integrate contextual information. In our approach, we also employ a transformer-based decoder, specifically designed to incorporate context in the token prediction process. We conduct a comprehensive examination of the influence of the encoder, decoder, noise scheduler, and self-conditioning on zero-shot generation. Furthermore, we compare TEncDM with previous approaches on three conditional text generation tasks: QQP, XSum, and Wiki-Auto. The results show that TEncDM exhibits superior performance compared to existing non-autoregressive diffusion models. Our code is available at https://github.com/M0RJIQUE/tencdm.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 16 of its 28 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

16 of 28 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model5 Jun
  2. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation16 May
  3. 2022Empowering Diffusion Models on the Embedding Space for Text Generation19 Dec
  4. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov

Other references · 12

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  8. 2022On Distillation of Guided Diffusion Modelscited by 721
  9. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  10. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  11. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  12. 2020Neural CRF Model for Sentence Alignment in Text Simplificationcited by 106

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 13.

22 papers.

Sort:

By area:

  1. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  2. 2025CANDI: Hybrid Discrete-Continuous Diffusion Modelscited by 31
  3. 2026ELF: Embedded Language Flowscited by 27
  4. 2025Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modelingcited by 24
  5. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generationcited by 19
  6. 2026Self-conditioned Flow Map Language Models via Fixed-point Flowscited by 9
  7. 2025Smoothie: Smoothing Diffusion on Token Embeddings for Text Generationcited by 8
  8. 2026How to Train Your Latent Diffusion Language Model Jointly With the Latent Spacecited by 8
  9. 2025Guided Star-Shaped Masked Diffusioncited by 5
  10. 2025Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processescited by 4
  11. 2026Revealing the Attention Floating Mechanism in Masked Diffusion Modelscited by 4
  12. 2026Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?cited by 3
10 more
  1. 2026DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modelingcited by 3
  2. 2026AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modelingcited by 3
  3. 2025Alternatives To Next Token Prediction In Text Generation -- A Surveycited by 1
  4. 2026Towards Latent Diffusion Suitable For Textcited by 1
  5. 2026Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Modelscited by 1
  6. 2025Latent-Augmented Discrete Diffusion Modelscited by 0
  7. 2026Improving Few-Step Language Flows with Untied Self-Conditioningcited by 0
  8. 2026Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Modelscited by 0
  9. 2026Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoderscited by 0
  10. 2026One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Role of Discreteness in Diffusion LLMs
  2. 2026Diffusion Language Models: An Experimental Analysis
  3. 2026Scaling and Distilling Text Embeddings for Better Diffusibility
  4. 2026One Token Is Enough: Improving Diffusion Language Models with a Sink Token
  5. 2025Theoretical Benefit and Limitation of Diffusion Language Model
  6. 2026Token Time Continuous Diffusion for Language Modeling
  7. 2026Rethinking Token Prediction: Tree-Structured Diffusion Language Model
  8. 2022Latent Diffusion for Language Generation