# Generative Frontiers: Why Evaluation Matters for Diffusion Language Models

arXiv: https://arxiv.org/abs/2604.02718 · HTML: https://mlsanity.com/paper/2604.02718/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 10 of its 12 references that are in the map.
- First posted on arXiv 3 Apr 2026.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 10 of 10)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models](https://mlsanity.com/paper/2503.09573/) (2025)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling](https://mlsanity.com/paper/2409.02908/) (2024)
- [The Diffusion Duality](https://mlsanity.com/paper/2506.10892/) (2025)
- [CANDI: Hybrid Discrete-Continuous Diffusion Models](https://mlsanity.com/paper/2510.22510/) (2025)

## Built on by (most cited first, 16 of 16)

- [Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation](https://mlsanity.com/paper/2605.22765/) (2026)
- [CoBit: Language Modeling with Bitstream Diffusion](https://mlsanity.com/paper/2605.07013/) (2026)
- [Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics](https://mlsanity.com/paper/2606.08417/) (2026)
- [Fixed-Point Masked Generative Modeling](https://mlsanity.com/paper/2605.31215/) (2026)
- [Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation](https://mlsanity.com/paper/2607.27372/) (2026)
- [TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Models](https://mlsanity.com/paper/2605.24292/) (2026)
- [Low Perplexity is Repetition: A One-Dimensional Self-Conditioning Attractor in Continuous Diffusion LMs](https://mlsanity.com/paper/2607.00588/) (2026)
- [CaRE Compute-aware Remasking Evaluation Protocol for Masked Diffusion Language Models](https://mlsanity.com/paper/2607.24763/) (2026)
- [Simplex Diffusion Models](https://mlsanity.com/paper/2609.35553/) (2026)
- [Flow Duality and Source Geometry for Categorical Generation](https://mlsanity.com/paper/2609.10863/) (2026)
- [How to Guide Your Language Flow](https://mlsanity.com/paper/2609.19356/) (2026)
- [Coherence-Aware Distributional Evaluation of Open-Ended Text Generation](https://mlsanity.com/paper/2609.34240/) (2026)
- [Distribution Matching Distillation for Continuous Diffusion Language Models](https://mlsanity.com/paper/2609.40235/) (2026)
- [Scaling and Distilling Text Embeddings for Better Diffusibility](https://mlsanity.com/paper/2610.01016/) (2026)
- [Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling](https://mlsanity.com/paper/2610.08738/) (2026)
- [Noise Your Prompt: Noising Conditioning Tokens in Continuous Diffusion Language Models](https://mlsanity.com/paper/2610.09145/) (2026)

## Abstract (arXiv)

Diffusion language models have seen exciting recent progress, offering far more flexibility in generative trajectories than autoregressive models. This flexibility has motivated a growing body of research into new approaches to diffusion language modeling, which typically begins at the scale of GPT-2 small (150 million parameters). However, these advances introduce new issues with evaluation methodology. In this technical note, we discuss the limitations of current methodology and propose principled augmentations to ensure reliable comparisons. We first discuss why OpenWebText has become the standard benchmark, and why alternatives such as LM1B are inherently less meaningful. We then discuss the limitations of likelihood evaluations for diffusion models, and explain why relying on generative perplexity alone as a metric can lead to uninformative results. To address this, we show that generative perplexity and entropy are two components of the KL divergence to a reference distribution. This decomposition explains generative perplexity's sensitivity to entropy, and naturally suggests generative frontiers as a principled method for evaluating model generative quality. We conclude with empirical observations on model quality at this scale. We include a blog post with interactive content to illustrate the argument at https://patrickpynadath1.github.io/blog/eval_methodology/.