mlsanityData to 13 Oct 2026

Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct

arXiv 2509.25035 ↗First posted 29 Sep 2025cs.CL cs.AI cs.LGcited by 16

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Fast and high-quality language generation is the holy grail that people pursue in the age of AI. In this work, we introduce Discrete Diffusion Divergence Instruct (DiDi-Instruct), a training-based method that initializes from a pre-trained diffusion large language model (dLLM) and distills a few-step student for fast generation. The model distilled with DiDi-Instruct matches or surpasses its dLLM teacher and the GPT-2 baseline while providing up to 64× acceleration. The theoretical foundation of DiDi-Instruct is a novel framework based on integral KL-divergence minimization, which leads to a practical training algorithm. We further introduce grouped reward normalization, intermediate-state matching, and the reward-guided ancestral sampler to improve training stability, model coverage, and inference quality. On the OpenWebText benchmark, DiDi-Instruct achieves perplexity ranging from 62.2 (8 NFEs) to 18.4 (128 NFEs), outperforming prior accelerated dLLMs and the GPT-2 baseline. These gains incur a negligible entropy loss (around 1%) and reduce additional training wall-clock time by more than 20× compared to competing dLLM distillation methods. We further validate the robustness and effectiveness of DiDi-Instruct through extensive ablation studies, model scaling, downstream task evaluations, and unconditional protein sequence generation. In conclusion, DiDi-Instruct enables efficient and effective distillation for language generation in the blink of an eye.

Key facts. Cited by 16 papers in the map (data through 13 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 95 of its 118 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

95 of 118 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Dream-Coder 7B: An Open Diffusion Language Model for Code1 Sep
  2. 2025Dream 7B: Diffusion Large Language Models21 Aug
  3. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference4 Aug
  4. 2025The Diffusion Duality12 Jun
  5. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  6. 2025Large Language Diffusion Models14 Feb
  7. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions10 Feb
  8. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time28 Oct
  9. 2024Think While You Generate: Discrete Diffusion with Planned Denoising8 Oct
  10. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  11. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  12. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
6 more
  1. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  2. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  3. 2023Likelihood-Based Diffusion Language Models30 May
  4. 2022Continuous diffusion for categorical data28 Nov
  5. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  6. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 77

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  5. 2023GPT-4 Technical Reportcited by 18,278
  6. 2017Proximal Policy Optimization Algorithmscited by 14,592
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  8. 2020Denoising Diffusion Implicit Modelscited by 9,718
  9. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,028
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,213
  12. 2020Measuring Massive Multitask Language Understandingcited by 7,876
38 more
  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,023
  2. 2022Scalable Diffusion Models with Transformerscited by 5,973
  3. 2022Generative Adversarial Networkscited by 5,614
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,628
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,607
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,197
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,023
  8. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,974
  9. 2023Consistency Modelscited by 1,797
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,685
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,625
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,201
  13. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,077
  14. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,012
  15. 2023One-step Diffusion with Distribution Matching Distillationcited by 991
  16. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 837
  17. 2023Adversarial Diffusion Distillationcited by 751
  18. 2022On Distillation of Guided Diffusion Modelscited by 721
  19. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 664
  20. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 570
  21. 2025Mean Flows for One-step Generative Modelingcited by 541
  22. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 529
  23. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 516
  24. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 468
  25. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusioncited by 454
  26. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 442
  27. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 438
  28. 2023Improved Techniques for Training Consistency Modelscited by 433
  29. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 409
  30. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 365
  31. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 347
  32. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 277
  33. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 275
  34. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 261
  35. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 234
  36. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 228
  37. 2024Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generationcited by 212
  38. 2022Diffusion-GAN: Training GANs with Diffusioncited by 204

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 3; 2026: 13.

16 papers.

Sort:

By area:

  1. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 17
  2. 2025dUltra: Ultra-Fast Diffusion Language Models via Reinforcement Learningcited by 10
  3. 2026IDLM: Inverse-distilled Diffusion Language Modelscited by 8
  4. 2026When to Commit? Towards Variable-Size Self-Contained Blocks for Discrete Diffusion Language Modelscited by 7
  5. 2026Coupling Models for One-Step Discrete Generationcited by 2
  6. 2026Drifting Objectives for Refining Discrete Diffusion Language Modelscited by 2
  7. 2026Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or Onecited by 2
  8. 2025One-step Diffusion Models with Bregman Density Ratio Matchingcited by 1
  9. 2026Ultra Fast PDE Solving via Physics Guided Few-step Diffusioncited by 1
  10. 2026Generalized Discrete Diffusion with Self-Correctioncited by 1
  11. 2026Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RLcited by 1
  12. 2026Commit Locally, Exit Globally: Coordinating Adaptive Sampling and Early Exit in Diffusion Language Modelscited by 0
4 more
  1. 2026Simple Diffusion Language Models Are More Effective Few-Step Generators Than Reportedcited by 0
  2. 2026Gumbel Straight Flow: Distilling Autoregressive Models into One-step Flow Mapscited by 0
  3. 2026IDRF: Inverse-Distilled Reward Fine-tuning of Masked Discrete Diffusion Modelscited by 0
  4. 2026Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
  2. 2026Improved Large Language Diffusion Models
  3. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  4. 2026Diffusion Language Models: An Experimental Analysis
  5. 2026Understanding and Accelerating the Training of Masked Diffusion Language Models
  6. 2026DiffusionGemma Technical Report
  7. 2022Latent Diffusion for Language Generation
  8. 2026Few-Step Diffusion Language Models via Trajectory Self-Distillation