mlsanityData to 7 Oct 2026

Distribution Matching Distillation for Continuous Diffusion Language Models

arXiv 2609.40235 ↗First posted 30 Sep 2026cs.LG cs.CL stat.MLcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Continuous diffusion language models generate all tokens in parallel, yet high-quality generation can still require hundreds of network evaluations (NFEs). We study how distributional distillation can reduce this cost by exploiting the student's probabilistic token outputs. Our unified formulation connects the student's output parameterization to the resulting gradient estimators and yields two methods with the same student architecture and reverse-KL matching objective: Simplex-DMD uses continuous token relaxations and pathwise gradients, while Reinforce-DMD uses categorical sampling and REINFORCE with a learned density ratio. We develop both methods for multi-step generation and investigate the training and sampling choices associated with each parameterization. On OpenWebText, for sequences of 1,024 tokens, Simplex-DMD achieves a generative perplexity of 45.6 at a unigram entropy of 5.44 nats in just 4 NFEs, a 49% reduction relative to the strongest evaluated diffusion baseline at matched entropy and sampling budget. Reinforce-DMD improves the frontier at larger budgets, reaching a generative perplexity of 14.9 at an entropy of 5.00 nats with 256 NFEs, a 20% reduction under the same comparison protocol.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 38 of its 45 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

38 of 45 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026Continuous Diffusion Scales Competitively with Discrete Diffusion for Language18 May
  2. 2026ELF: Embedded Language Flows11 May
  3. 2026Scaling Categorical Flow Maps8 May
  4. 2026LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling13 Apr
  5. 2026Generative Frontiers: Why Evaluation Matters for Diffusion Language Models3 Apr
  6. 2026IDLM: Inverse-distilled Diffusion Language Models22 Feb
  7. 2026Flow Map Language Models: One-step Language Modeling via Continuous Denoising18 Feb
  8. 2025The Diffusion Duality12 Jun
  9. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generation30 May
  10. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time28 Oct
  11. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  12. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
3 more
  1. 2023Likelihood-Based Diffusion Language Models30 May
  2. 2022Continuous diffusion for categorical data28 Nov
  3. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov

Other references · 23

  1. 2020Denoising Diffusion Implicit Modelscited by 9,655
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  4. 2023Consistency Modelscited by 1,778
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  6. 2021Variational Diffusion Modelscited by 1,145
  7. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  8. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  9. 2023Improved Techniques for Training Consistency Modelscited by 430
  10. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  11. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 232
  12. 2024Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generationcited by 211
11 more
  1. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 148
  2. 2024Flow map matching with stochastic interpolants: A mathematical framework for consistency modelscited by 99
  3. 2025How to build a consistency model: Learning flow maps via self-distillationcited by 96
  4. 2024Multistep Distillation of Diffusion Models via Moment Matchingcited by 76
  5. 2024Distillation of Discrete Diffusion through Dimensional Correlationscited by 46
  6. 2026Categorical Flow Mapscited by 37
  7. 2024Flow Generator Matchingcited by 26
  8. 2026Discrete Flow Mapscited by 23
  9. 2025ReDi: Rectified Discrete Flowcited by 13
  10. 2026Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulationcited by 11
  11. 2026Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMDcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Adaptation to Intrinsic Dependence in Diffusion Language Models
  2. 2026CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think
  3. 2026A Tale of Two Temperatures: Simple, Efficient, and Diverse Sampling from Diffusion Language Models
  4. 2026MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Scaling of Diffusion Language Models
  5. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models
  6. 2026Token Time Continuous Diffusion for Language Modeling
  7. 2025Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models
  8. 2026Simple Diffusion Language Models Are More Effective Few-Step Generators Than Reported