mlsanityData to 14 Oct 2026

Conditional [MASK] Discrete Diffusion Language Model

arXiv 2411.06438 ↗First posted 10 Nov 2024cs.CL cs.AIcited by 5

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Although auto-regressive models excel in natural language processing, they often struggle to generate diverse text and provide limited controllability. Non-auto-regressive methods could be an alternative but often produce degenerate outputs and exhibit shortcomings in conditional generation. To address these challenges, we propose Diffusion-EAGS, a novel framework that integrates conditional masked language models into diffusion language models through the theoretical lens of a conditional Markov Random Field. In doing so, we propose entropy-adaptive Gibbs sampling and entropy-based noise scheduling to counterbalance each model's shortcomings. Experimental results show that Diffusion-EAGS outperforms baselines and achieves the best quality-diversity tradeoff, demonstrating its effectiveness in non-autoregressive text generation.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 53 of its 80 references that are in the map. First posted on arXiv 10 Nov 2024.

Builds on

53 of 80 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Energy-Based Diffusion Language Models for Text Generation28 Oct
  2. 2024Scaling up Masked Diffusion Models on Text24 Oct
  3. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning18 Oct
  4. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  5. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  6. 2024DiffusionDialog: A Diffusion Model for Diverse Dialog Generation with Latent Space10 Apr
  7. 2024Language Rectified Flow: Advancing Diffusion Language Generation with Probabilistic Flows25 Mar
  8. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  9. 2023DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Models9 Oct
  10. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation16 May
  11. 2023Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation6 May
  12. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noise10 Apr
5 more
  1. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises20 Feb
  2. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise22 Dec
  3. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers20 Dec
  4. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  5. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 36

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  10. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  12. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
24 more
  1. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  2. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  3. 2020Aligning AI With Shared Human Valuescited by 723
  4. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  5. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  6. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  7. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  8. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  9. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  10. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 311
  11. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  12. 2022Cold Diffusion: Inverting Arbitrary Image Transforms Without Noisecited by 276
  13. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  14. 2024Chain-of-Thought Reasoning Without Promptingcited by 210
  15. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 185
  16. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 154
  17. 2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 121
  18. 2019FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flowcited by 89
  19. 2024ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMcited by 73
  20. 2024Amortizing intractable inference in diffusion models for vision, language, and controlcited by 72
  21. 2017Quasar: Datasets for Question Answering by Search and Readingcited by 50
  22. 2021Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastingscited by 37
  23. 2022Causes and Cures for Interference in Multilingual Translationcited by 23
  24. 2024LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation?cited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Where-to-Unmask: Ground-Truth-Guided Unmasking Order Learning for Masked Diffusion Language Modelscited by 9
  2. 2026STDec: Spatio-Temporal Stability Guided Decoding for dLLMscited by 1
  3. 2026When to Plan, When to Polish: Noise Level as a Granularity Axis for Diffusion Language Modelscited by 0
  4. 2026Discretizing Continuous Time Series for Imputation with Masked Diffusion Trainingcited by 0
  5. 2026CoDR: Training-Free Confidence-Drift Remasking for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Simple and Effective Masked Diffusion Language Models
  2. 2025Theoretical Benefit and Limitation of Diffusion Language Model
  3. 2026Multi-Mask Diffusion Language Models for Few-Step Generation
  4. 2026CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language
  5. 2026Auto-Regressive Masked Diffusion Models
  6. 2025Plan for Speed: Dilated Scheduling for Masked Diffusion Language Models
  7. 2025Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policies
  8. 2026Diffusion Language Models: An Experimental Analysis