mlsanityData to 14 Oct 2026

Discrete Diffusion in Large Language and Multimodal Models: A Survey

arXiv 2506.13759 ↗First posted 16 Jun 2025cs.LG cs.AIcited by 61

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

also in 1 more
Abstract (arXiv)

In this work, we provide a systematic survey of Discrete Diffusion Language Models (dLLMs) and Discrete Diffusion Multimodal Language Models (dMLLMs). Unlike autoregressive (AR) models, dLLMs and dMLLMs adopt a multi-token, parallel decoding paradigm using full attention and a denoising-based generation strategy. This paradigm naturally enables parallel generation, fine-grained output control, and dynamic perception. These capabilities are previously difficult to achieve with AR models. A growing number of industrial-scale proprietary d(M)LLMs, as well as a large number of open-source academic d(M)LLMs, have demonstrated performance comparable to their autoregressive counterparts, while achieving up to 10× acceleration in inference speed. These developments position discrete diffusion models as a promising alternative to intelligence based on the traditional autoregressive approach. In this work, we present a comprehensive overview of the research in the dLLM and dMLLM domains. We trace the historical development of dLLMs and dMLLMs, formalize the underlying mathematical frameworks, list commonly-used modeling methods, and categorize representative models. We further analyze key techniques for training, inference, quantization. We also discuss the trustworthy issues and summarize emerging applications across language, vision-language, and biological domains and etc.. We conclude by discussing future directions for research and deployment. Relative papers are collected in https://github.com/LiQiiiii/Awesome-Discrete-Diffusion-LLMMLLM

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 122 of its 180 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

122 of 180 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding22 May
  2. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understanding22 May

Other references · 120

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2025Qwen3 Technical Reportcited by 8,820
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
38 more
  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2016Deep Learning with Differential Privacycited by 2,728
  13. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  14. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  15. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  16. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  17. 2025Large Language Diffusion Modelscited by 912
  18. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  19. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  20. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  21. 2023Extracting Training Data from Diffusion Modelscited by 732
  22. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  23. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  24. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  25. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  26. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  27. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  28. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  29. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  30. 2021Explaining Neural Scaling Lawscited by 363
  31. 2024Discrete Flow Matchingcited by 317
  32. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  33. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274
  34. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  35. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  36. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  37. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 214
  38. 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 210

50 most cited of 120 All 120 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 21; 2026: 40.

61 papers.

Sort:

By area:

  1. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 166
  2. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 129
  3. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 65
  4. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 34
  5. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 23
  6. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 18
  7. 2025Unveiling the Potential of Diffusion Large Language Model in Controllable Generationcited by 15
  8. 2025SDAR-VL: Stable and Efficient Block-wise Diffusion for Vision-Language Understandingcited by 13
  9. 2025DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Modelscited by 12
  10. 2025Rainbow Padding: Mitigating Early Termination in Instruction-Tuned Diffusion LLMscited by 10
  11. 2025WAM-Diff: A Masked Diffusion VLA Framework with MoE and Online Reinforcement Learning for Autonomous Drivingcited by 10
  12. 2025A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Modelscited by 9
38 more
  1. 2026Fast-dVLA: Accelerating Discrete Diffusion VLA to Real-Time Performancecited by 9
  2. 2026Make Geometry Matter for Spatial Reasoningcited by 8
  3. 2026On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignmentcited by 8
  4. 2026When to Commit? Towards Variable-Size Self-Contained Blocks for Discrete Diffusion Language Modelscited by 7
  5. 2026ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skippingcited by 6
  6. 2026dOPSD: On-Policy Self-Distillation for Diffusion Language Modelscited by 6
  7. 2025From Masks to Worlds: A Hitchhiker's Guide to World Modelscited by 5
  8. 2025Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Datacited by 5
  9. 2026DFM-VLA: Iterative Action Refinement for Robot Manipulation via Discrete Flow Matchingcited by 5
  10. 2025DMark: Order-Agnostic Watermarking for Diffusion Large Language Modelscited by 4
  11. 2025Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMscited by 4
  12. 2025HD-Prot: A Protein Language Model for Joint Sequence-Structure Modeling with Continuous Structure Tokenscited by 4
  13. 2026ILRR: Inference-Time Steering Method for Masked Diffusion Language Modelscited by 4
  14. 2026Breaking Block Boundaries: Anchor-based History-stable Decoding for Diffusion Large Language Modelscited by 4
  15. 2026Simple Self-Conditioning Adaptation for Masked Diffusion Modelscited by 4
  16. 2025Planner and Executor: Collaboration between Discrete Diffusion And Autoregressive Models in Reasoningcited by 3
  17. 2025Masked Diffusion Models are Secretly Learned-Order Autoregressive Modelscited by 3
  18. 2025Guided Transfer Learning for Discrete Diffusion Modelscited by 3
  19. 2026LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Modelscited by 3
  20. 2026Neither Parallel Nor Sequential: How DiffusionGemma Actually Commits Tokenscited by 3
  21. 2026Mosaic: Unlocking Long-Context Inference for Diffusion LLMs via Global Memory Planning and Dynamic Peak Tamingcited by 2
  22. 2026DiMo: Discrete Diffusion Modeling for Motion Generation and Understandingcited by 2
  23. 2026Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Modelscited by 2
  24. 2026Reasoning on the Manifold: Bidirectional Consistency for Self-Verification in Diffusion Language Modelscited by 2
  25. 2026dMoE: dLLMs with Learnable Block Expertscited by 2
  26. 2026dgMARK: Decoding-Guided Watermarking for Diffusion Language Modelscited by 1
  27. 2026Step-Wise Refusal Dynamics in Autoregressive and Diffusion Language Modelscited by 1
  28. 2026VDLM: Variable Diffusion LMs via Robust Latent-to-Text Renderingcited by 1
  29. 2026DepCap: Adaptive Block-Wise Parallel Decoding for Efficient Diffusion LM Inferencecited by 1
  30. 2026Yeti: A compact protein structure tokenizer for reconstruction and multi-modal generationcited by 1
  31. 2026ViMU: Benchmarking Video Metaphorical Understandingcited by 1
  32. 2026Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamicscited by 1
  33. 2025Syntax-Guided Diffusion Language Models with User-Integrated Personalizationcited by 0
  34. 2025STaRR: Spatial-Temporal Token-Dynamics-Aware Responsive Remasking for Diffusion Language Modelscited by 0
  35. 2026Efficient Self-Evaluation for Diffusion Language Models via Sequence Regenerationcited by 0
  36. 2026Controllable Accent Normalization via Discrete Diffusioncited by 0
  37. 2026AutoMIA: Improved Baselines for Membership Inference Attack via Agentic Self-Explorationcited by 0
  38. 2026Reinforcement Learning from Denoising Feedbackcited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Diffusion Language Models
  2. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing
  3. 2025DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models
  4. 2026Diffusion Language Models: An Experimental Analysis
  5. 2026Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLM
  6. 2026dLLM: Simple Diffusion Language Modeling
  7. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  8. 2026Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models