mlsanityData to 14 Oct 2026

The Diffusion Duality

arXiv 2506.10892 ↗First posted 12 Jun 2025cs.LG cs.AI cs.CLcited by 135

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Uniform-state discrete diffusion models hold the promise of fast text generation due to their inherent ability to self-correct. However, they are typically outperformed by autoregressive models and masked diffusion models. In this work, we narrow this performance gap by leveraging a key insight: Uniform-state diffusion processes naturally emerge from an underlying Gaussian diffusion. Our method, Duo, transfers powerful techniques from Gaussian diffusion to improve both training and sampling. First, we introduce a curriculum learning strategy guided by the Gaussian process, doubling training speed by reducing variance. Models trained with curriculum learning surpass autoregressive models in zero-shot perplexity on 3 of 7 benchmarks. Second, we present Discrete Consistency Distillation, which adapts consistency distillation from the continuous to the discrete setting. This algorithm unlocks few-step generation in diffusion language models by accelerating sampling by two orders of magnitude. We provide the code, model checkpoints, and video tutorials on the project page: http://s-sahoo.github.io/duo

Key facts. Cited by 135 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 55 of its 77 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

55 of 77 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  2. 2025Remasking Discrete Diffusion Models with Inference-Time Scaling1 Mar
  3. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time28 Oct
  4. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  5. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  6. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  7. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  8. 2023Likelihood-Based Diffusion Language Models30 May
  9. 2022Continuous diffusion for categorical data28 Nov
  10. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  11. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct

Other references · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Scalable Diffusion Models with Transformerscited by 6,054
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
32 more
  1. 2022Video Diffusion Modelscited by 2,305
  2. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  4. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  5. 2021Variational Diffusion Modelscited by 1,153
  6. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  7. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  8. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  10. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  11. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  12. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  13. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  14. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  15. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  16. 2023Improved Techniques for Training Consistency Modelscited by 439
  17. 2024One Step Diffusion via Shortcut Modelscited by 391
  18. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
  19. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 348
  20. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  21. 2019Differentiation of Blackbox Combinatorial Solverscited by 226
  22. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 185
  23. 2018Learning Latent Permutations with Gumbel-Sinkhorn Networkscited by 175
  24. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 155
  25. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 129
  26. 2023Generative Diffusion Models on Graphs: Methods and Applicationscited by 81
  27. 2024Informed Correctors for Discrete Diffusion Modelscited by 57
  28. 2023Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEscited by 54
  29. 2023Diffusion Models With Learned Adaptive Noisecited by 51
  30. 2025GenMol: A Drug Discovery Generalist with Discrete Diffusioncited by 50
  31. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 34
  32. 2022Backpropagation through Combinatorial Algorithms: Identity with Projection Workscited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 31; 2026: 104.

135 papers.

Sort:

By area:

  1. 2025A Survey on Diffusion Language Modelscited by 78
  2. 2026ELF: Embedded Language Flowscited by 45
  3. 2026Categorical Flow Mapscited by 44
  4. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 41
  5. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 37
  6. 2026LangFlow: Continuous Diffusion Rivals Discrete in Language Modelingcited by 36
  7. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 35
  8. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 34
  9. 2025CANDI: Hybrid Discrete-Continuous Diffusion Modelscited by 34
  10. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 33
  11. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasonercited by 32
  12. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
38 more
  1. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimationcited by 27
  2. 2026Scaling Beyond Masked Diffusion Language Modelscited by 27
  3. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 24
  4. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learningcited by 19
  5. 2026Continuous Diffusion Scales Competitively with Discrete Diffusion for Languagecited by 19
  6. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 18
  7. 2025Ultra-Fast Language Generation via Discrete Diffusion Divergence Instructcited by 16
  8. 2025Soft-Masked Diffusion Language Modelscited by 16
  9. 2026Generative Frontiers: Why Evaluation Matters for Diffusion Language Modelscited by 16
  10. 2025ReDi: Rectified Discrete Flowcited by 14
  11. 2026Continuous Latent Diffusion Language Modelcited by 14
  12. 2026Self-conditioned Flow Map Language Models via Fixed-point Flowscited by 14
  13. 2026Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulationcited by 13
  14. 2026The Diffusion Duality, Chapter II: Psi-Samplerscited by 12
  15. 2026Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMDcited by 12
  16. 2025The Cosine Schedule is Fisher-Rao-Optimal for Masked Discrete Diffusion Modelscited by 11
  17. 2025Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedulescited by 11
  18. 2026How to Train Your Latent Diffusion Language Model Jointly With the Latent Spacecited by 11
  19. 2025Watermarking Diffusion Language Modelscited by 10
  20. 2025Next Semantic Scale Prediction via Hierarchical Diffusion Language Modelscited by 8
  21. 2025Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wallcited by 8
  22. 2026IDLM: Inverse-distilled Diffusion Language Modelscited by 8
  23. 2026Scaling Categorical Flow Mapscited by 8
  24. 2025Diffusion and Flow-based Copulas: Forgetting and Remembering Dependenciescited by 7
  25. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  26. 2026Prism: Efficient Test-Time Scaling via Hierarchical Search and Self-Verification for Discrete Diffusion Language Modelscited by 7
  27. 2026Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metricscited by 7
  28. 2025Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddingscited by 6
  29. 2025Data-to-Energy Stochastic Dynamicscited by 6
  30. 2025DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbonecited by 6
  31. 2026Unifying Masked Diffusion Models with Various Generation Orders and Beyondcited by 6
  32. 2026Spherical Flows for Sampling Categorical Datacited by 6
  33. 2026FLARE: Diffusion for Hybrid Language Modelcited by 6
  34. 2025PairFlow: Closed-Form Source-Target Coupling for Few-Step Generation in Discrete Flow Modelscited by 5
  35. 2026Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Modelscited by 5
  36. 2026Fixed-Point Masked Generative Modelingcited by 5
  37. 2026AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modelingcited by 5
  38. 2025Efficient Autoregressive Inference for Transformer Probabilistic Modelscited by 4

50 most cited of 135 All 135 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Role of Discreteness in Diffusion LLMs
  2. 2026Parallelism, critical windows, and separations among diffusion language models
  3. 2022Self-conditioned Embedding Diffusion for Text Generation
  4. 2026Consistent Diffusion Language Models
  5. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models
  6. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models
  7. 2023Can Diffusion Model Achieve Better Performance in Text Generation? Bridging the Gap between Training and Inference!
  8. 2025Discrete Diffusion Models for Language Generation