mlsanityData to 7 Oct 2026

InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation

arXiv 2310.11976 ↗First posted 18 Oct 2023cs.CL cs.AIcited by 5

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion models have garnered considerable interest in the field of text generation. Several studies have explored text diffusion models with different structures and applied them to various tasks, including named entity recognition and summarization. However, there exists a notable disparity between the "easy-first" text generation process of current diffusion models and the "keyword-first" natural text generation process of humans, which has received limited attention. To bridge this gap, we propose InfoDiffusion, a non-autoregressive text diffusion model. Our approach introduces a "keyinfo-first" generation strategy and incorporates a noise schedule based on the amount of text information. In addition, InfoDiffusion combines self-conditioning with a newly proposed partially noising model structure. Experimental results show that InfoDiffusion outperforms the baseline model in terms of generation quality and diversity, as well as exhibiting higher sampling efficiency.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 34 of its 45 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

34 of 45 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation6 May
  2. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noise10 Apr
  3. 2023Diffusion Models for Non-autoregressive Text Generation: A Survey12 Mar
  4. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises20 Feb
  5. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers20 Dec
  6. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  7. 2022DiffusER: Discrete Diffusion via Edit-based Reconstruction30 Oct
  8. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
14 more
  1. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  2. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  3. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  4. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  5. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  6. 2020Neural CRF Model for Sentence Alignment in Text Simplificationcited by 106
  7. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  8. 2018Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speechcited by 55
  9. 2017Quasar: Datasets for Question Answering by Search and Readingcited by 50
  10. 2023DiffuSum: Generation Enhanced Extractive Summarization with Diffusioncited by 25
  11. 2022Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priorscited by 13
  12. 2019Sequence Modeling with Unconstrained Generation Ordercited by 8
  13. 2023CTRLStruct: Dialogue Structure Learning for Open-Domain Response Generationcited by 8
  14. 2023Unified Text Structuralization with Instruction-tuned Language Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2025A Survey on Diffusion Language Modelscited by 75
  2. 2026ELF: Embedded Language Flowscited by 27
  3. 2025Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMscited by 4
  4. 2024Semantic are Beacons: A Semantic Perspective for Unveiling Parameter-Efficient Fine-Tuning in Knowledge Learningcited by 3
  5. 2026Unlocking the Potentials of Retrieval-Augmented Generation for Diffusion Language Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023A Reparameterized Discrete Diffusion Model for Text Generation
  2. 2022Self-conditioned Embedding Diffusion for Text Generation
  3. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model
  4. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
  5. 2025Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes
  6. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusion
  7. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control
  8. 2023A Survey of Diffusion Models in Natural Language Processing