mlsanityData to 7 Oct 2026

Distillation of Discrete Diffusion through Dimensional Correlations

arXiv 2410.08709 ↗First posted 11 Oct 2024cs.LG cs.NA math.NAcited by 46

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models have demonstrated exceptional performances in various fields of generative modeling, but suffer from slow sampling speed due to their iterative nature. While this issue is being addressed in continuous domains, discrete diffusion models face unique challenges, particularly in capturing dependencies between elements (e.g., pixel relationships in image, sequential dependencies in language) mainly due to the computational cost of processing high-dimensional joint distributions. In this paper, (i) we propose "mixture" models for discrete diffusion that are capable of treating dimensional correlations while remaining scalable, and (ii) we provide a set of loss functions for distilling the iterations of existing models. Two primary theoretical insights underpin our approach: First, conventional models with element-wise independence can well approximate the data distribution, but essentially require it many sampling steps. Second, our loss functions enable the mixture models to distill such many-step conventional models into just a few steps by learning the dimensional correlations. Our experimental results show the effectiveness of the proposed method in distilling pretrained discrete diffusion models across image and language domains. The code used in the paper is available at https://github.com/sony/di4c .

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 53 of its 75 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

53 of 75 references.

Closest ancestors · 26

Same direction, newest first

  1. 2024textitJump Your Steps: Optimizing Sampling Schedule of Discrete Diffusion Models10 Oct
  2. 2023Improved Techniques for Training Consistency Models22 Oct
  3. 2023DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics20 Oct
  4. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion1 Oct
  5. 2023Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models25 Sep
  6. 2023On the Equivalence of Consistency-Type Models: Consistency Models, Consistent Diffusion Models, and Fokker-Planck Regularization1 Jun
  7. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model17 Mar
  8. 2023Consistency Models2 Mar
  9. 2023Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent17 Feb
  10. 2023Universal Guidance for Diffusion Models14 Feb
  11. 2022Score-based Continuous-time Discrete Diffusion Models30 Nov
  12. 2022Fast Sampling of Diffusion Models via Operator Learning24 Nov
14 more
  1. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models2 Nov
  2. 2022On Distillation of Guided Diffusion Models6 Oct
  3. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  4. 2022Improved Vector Quantized Diffusion Models31 May
  5. 2022A Continuous Time Framework for Discrete Denoising Models30 May
  6. 2022Fast Sampling of Diffusion Models with Exponential Integrator29 Apr
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  8. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANs15 Dec
  9. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesis29 Nov
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  11. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed7 Jan
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  13. 2020Denoising Diffusion Implicit Models6 Oct
  14. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 27

  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  7. 2022Video Diffusion Modelscited by 2,280
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
  10. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  11. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  12. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
15 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  3. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  4. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  5. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  6. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  7. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  8. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  9. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  10. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  11. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  12. 2014Control functionals for Monte Carlo integrationcited by 59
  13. 2024Discrete Copula Diffusioncited by 53
  14. 2023A Pytorch Reproduction of Masked Generative Image Transformercited by 25
  15. 2024SpecMaskGIT: Masked Generative Modeling of Audio Spectrograms for Efficient Audio Synthesis and Beyondcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 29.

46 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 872
  2. 2025A Survey on Diffusion Language Modelscited by 75
  3. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  4. 2025Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithmscited by 47
  5. 2025KLASS: KL-Guided Fast Inference in Masked Diffusion Modelscited by 44
  6. 2026Categorical Flow Mapscited by 37
  7. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Modelscited by 23
  8. 2026Discrete Flow Mapscited by 23
  9. 2025Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Maskingcited by 19
  10. 2025Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modelingcited by 14
  11. 2025ReDi: Rectified Discrete Flowcited by 13
  12. 2025Self-Speculative Masked Diffusionscited by 11
34 more
  1. 2026Breaking the Factorization Barrier in Diffusion Language Modelscited by 11
  2. 2025Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusioncited by 10
  3. 2025How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practicescited by 10
  4. 2026Self-conditioned Flow Map Language Models via Fixed-point Flowscited by 9
  5. 2026DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmaskingcited by 8
  6. 2026Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMDcited by 8
  7. 2025Partition Generative Modeling: Masked Modeling Without Maskscited by 6
  8. 2026Scaling Categorical Flow Mapscited by 6
  9. 2025Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generatorcited by 5
  10. 2025Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddingscited by 5
  11. 2026IDLM: Inverse-distilled Diffusion Language Modelscited by 5
  12. 2025PairFlow: Closed-Form Source-Target Coupling for Few-Step Generation in Discrete Flow Modelscited by 4
  13. 2026Discrete diffusion samplers and bridges: Off-policy algorithms and applications in latent spacescited by 2
  14. 2026Simplex Relaxation for Discrete Diffusioncited by 2
  15. 2025UniCMs: A Unified Consistency Model For Efficient Multimodal Generation and Understandingcited by 1
  16. 2026Infinite Mask Diffusion for Few-Step Distillationcited by 1
  17. 2026One-Step Distillation of Discrete Diffusion Image Generators via Fixed-Point Iterationcited by 1
  18. 2026Learned Relay Representations for Forward-Thinking Discrete Diffusion Modelscited by 1
  19. 2026Tensor-Train Joint Modeling for Few-Step Discrete Diffusioncited by 1
  20. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniquescited by 1
  21. 2026Expanding Flow Mapscited by 1
  22. 2026Latent-Kernel Discrete Flow Maps for Few-Step Generationcited by 1
  23. 2026From Truncation to Commitment: Persistent Context in Uniform Discrete Diffusioncited by 1
  24. 2026Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Modelscited by 0
  25. 2026Understanding and Accelerating the Training of Masked Diffusion Language Modelscited by 0
  26. 2026Drifting Objectives for Refining Discrete Diffusion Language Modelscited by 0
  27. 2026On the Error-Correcting Effects of Stochasticity in Discrete Diffusioncited by 0
  28. 2026A Ticket from Marginals to Joints: Coupled-Noise Distillation for One-Step Block Generation in Diffusion Language Modelscited by 0
  29. 2026Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Samplingcited by 0
  30. 2026Simple Diffusion Language Models Are More Effective Few-Step Generators Than Reportedcited by 0
  31. 2026Acceleration of Diffusion Language Model through Discrete Average Generatorcited by 0
  32. 2026Distribution Matching Distillation for Continuous Diffusion Language Modelscited by 0
  33. 2026Gumbel Straight Flow: Distilling Autoregressive Models into One-step Flow Mapscited by 0
  34. 2026Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents
  2. 2025Distillation of Discrete Diffusion by Exact Conditional Distribution Matching
  3. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models
  4. 2024DDIL: Diversity Enhancing Diffusion Distillation With Imitation Learning
  5. 2024Relational Diffusion Distillation for Efficient Image Generation
  6. 2025A Survey on Pre-Trained Diffusion Model Distillations
  7. 2024Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions
  8. 2024Discrete generative diffusion models without stochastic differential equations: a tensor network approach