# Discrete Diffusion Models for Language Generation

arXiv: https://arxiv.org/abs/2507.07050 · HTML: https://mlsanity.com/paper/2507.07050/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 7 of its 19 references that are in the map.
- First posted on arXiv 2 Jul 2025.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 7 of 7)

- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [What is Wrong with Perplexity for Long-context Language Modeling?](https://mlsanity.com/paper/2410.23771/) (2024)
- [MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts](https://mlsanity.com/paper/2311.09580/) (2023)
- [Fast Generation for Convolutional Autoregressive Models](https://mlsanity.com/paper/1704.06001/) (2017)
- [Better Character Language Modeling Through Morphology](https://mlsanity.com/paper/1906.01037/) (2019)
- [A Survey on Prompting Techniques in LLMs](https://mlsanity.com/paper/2312.03740/) (2023)
- [Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives](https://mlsanity.com/paper/2309.00380/) (2023)

## Built on by (most cited first, 2 of 2)

- [LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning](https://mlsanity.com/paper/2510.04573/) (2025)
- [Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding](https://mlsanity.com/paper/2508.19529/) (2025)

## Abstract (arXiv)

Diffusion models have emerged as a powerful class of generative models, achieving state-of-the-art results in continuous data domains such as image and video generation. Their core mechanism involves a forward diffusion process that gradually transforms structured data into a Gaussian-like distribution, followed by a learned reverse process to reconstruct the data. While successful in continuous modalities, applying this framework to discrete data-particularly natural language-remains challenging due to token dependency complexities and the lack of a defined generation order.This thesis investigates the feasibility and performance of discrete diffusion models for natural language generation. Specifically, we evaluate the Discrete Denoising Diffusion Probabilistic Model (D3PM) and compare it with traditional autoregressive (AR) language models. To assess generative performance, we use Bits Per Token (BPT), Negative Log-Likelihood (NLL), Perplexity (PPL), and Batch Processing Speed. Results show the best-performing D3PM model achieves a BPT of 5.72, with a mean of 8.05. The AR model outperforms in compression with a lower mean BPT of 4.59, but D3PM achieves higher processing speed, reaching up to 3.97 batches per sec., indicating potential for parallel generation.All evaluations were conducted under consistent conditions-generating 100,000 tokens per model with a fixed batch size of four-for fair comparison. This research presents a detailed analysis of diffusion-based vs. autoregressive models, highlighting trade-offs in generative quality and efficiency. Findings emphasize both the promise and limitations of diffusion models for discrete data, supporting future work in non-autoregressive language generation.