# InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation

arXiv: https://arxiv.org/abs/2310.11976 · HTML: https://mlsanity.com/paper/2310.11976/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 18 Oct 2023.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022)
- [Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning](https://mlsanity.com/paper/2208.04202/) (2022)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models](https://mlsanity.com/paper/2211.15029/) (2022)
- [Neural CRF Model for Sentence Alignment in Text Simplification](https://mlsanity.com/paper/2005.02324/) (2020)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers](https://mlsanity.com/paper/2212.10325/) (2022)
- [Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech](https://mlsanity.com/paper/1805.12589/) (2018)
- [Quasar: Datasets for Question Answering by Search and Reading](https://mlsanity.com/paper/1707.03904/) (2017)
- [DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises](https://mlsanity.com/paper/2302.10025/) (2023)
- [Diffusion Models for Non-autoregressive Text Generation: A Survey](https://mlsanity.com/paper/2303.06574/) (2023)
- [DiffusER: Discrete Diffusion via Edit-based Reconstruction](https://mlsanity.com/paper/2210.16886/) (2022)
- [DiffuSum: Generation Enhanced Extractive Summarization with Diffusion](https://mlsanity.com/paper/2305.01735/) (2023)
- [A Cheaper and Better Diffusion Language Model with Soft-Masked Noise](https://mlsanity.com/paper/2304.04746/) (2023)
- [Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation](https://mlsanity.com/paper/2305.04044/) (2023)

## Built on by (most cited first, 5 of 5)

- [A Survey on Diffusion Language Models](https://mlsanity.com/paper/2508.10875/) (2025)
- [ELF: Embedded Language Flows](https://mlsanity.com/paper/2605.10938/) (2026)
- [Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs](https://mlsanity.com/paper/2506.05387/) (2025)
- [Semantic are Beacons: A Semantic Perspective for Unveiling Parameter-Efficient Fine-Tuning in Knowledge Learning](https://mlsanity.com/paper/2405.18292/) (2024)
- [Unlocking the Potentials of Retrieval-Augmented Generation for Diffusion Language Models](https://mlsanity.com/paper/2601.11342/) (2026)

## Abstract (arXiv)

Diffusion models have garnered considerable interest in the field of text generation. Several studies have explored text diffusion models with different structures and applied them to various tasks, including named entity recognition and summarization. However, there exists a notable disparity between the "easy-first" text generation process of current diffusion models and the "keyword-first" natural text generation process of humans, which has received limited attention. To bridge this gap, we propose InfoDiffusion, a non-autoregressive text diffusion model. Our approach introduces a "keyinfo-first" generation strategy and incorporates a noise schedule based on the amount of text information. In addition, InfoDiffusion combines self-conditioning with a newly proposed partially noising model structure. Experimental results show that InfoDiffusion outperforms the baseline model in terms of generation quality and diversity, as well as exhibiting higher sampling efficiency.