mlsanityData to 7 Oct 2026

Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD

arXiv 2603.20155 ↗First posted 20 Mar 2026cs.LG cs.CV stat.MLcited by 8

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

It is currently difficult to distill discrete diffusion models. In contrast, continuous diffusion literature has many distillation approaches methods that can reduce sampling steps to a handful. Our method, Discrete Moment Matching Distillation (D-MMD), leverages ideas that have been highly successful in the continuous domain. Whereas previous discrete distillation methods collapse, D-MMD maintains high quality and diversity (given sufficient sampling steps). This is demonstrated on both text and image datasets. Moreover, the newly distilled generators can outperform their teachers.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 28 of its 34 references that are in the map. First posted on arXiv 20 Mar 2026.

Builds on

28 of 34 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models14 Oct
  2. 2024Distillation of Discrete Diffusion through Dimensional Correlations11 Oct
  3. 2024Multistep Distillation of Diffusion Models via Moment Matching6 Jun
  4. 2024Multistep Consistency Models11 Mar
  5. 2023One-step Diffusion with Distribution Matching Distillation30 Nov
  6. 2023Improved Techniques for Training Consistency Models22 Oct
  7. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion1 Oct
  8. 2023Consistency Models2 Mar
  9. 2022On Distillation of Guided Diffusion Models6 Oct
  10. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning8 Aug
  11. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
1 more
  1. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 15

  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  4. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  5. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  6. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  7. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  8. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  9. 2024Discrete Flow Matchingcited by 301
  10. 2020Evaluation of Text Generation: A Surveycited by 275
  11. 2025The Diffusion Dualitycited by 124
  12. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Timecited by 73
3 more
  1. 2026Flow Map Language Models: One-step Language Modeling via Continuous Denoisingcited by 49
  2. 2026Categorical Flow Mapscited by 37
  3. 2026IDLM: Inverse-distilled Diffusion Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metricscited by 6
  2. 2026Fixed-Point Masked Generative Modelingcited by 2
  3. 2026One-Step Distillation of Discrete Diffusion Image Generators via Fixed-Point Iterationcited by 1
  4. 2026TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Modelscited by 1
  5. 2026Multi-Mask Diffusion Language Models for Few-Step Generationcited by 1
  6. 2026Simplex Diffusion Modelscited by 1
  7. 2026Distribution Matching Distillation for Continuous Diffusion Language Modelscited by 0
  8. 2026Gumbel Straight Flow: Distilling Autoregressive Models into One-step Flow Mapscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scale-wise Distillation of Diffusion Models
  2. 2026Continuous-Time Distribution Matching for Few-Step Diffusion Distillation
  3. 2025Distillation of Discrete Diffusion by Exact Conditional Distribution Matching
  4. 20261.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation
  5. 2026Consistent Distribution Matching for Data-Free Diffusion Distillation
  6. 2024Relational Diffusion Distillation for Efficient Image Generation
  7. 2025Learnable Sampler Distillation for Discrete Diffusion Models
  8. 2025Discrete Spatial Diffusion: Intensity-Preserving Diffusion Modeling