mlsanityData to 14 Oct 2026

dKV-Cache: The Cache for Diffusion Language Models

arXiv 2505.15781 ↗First posted 21 May 2025cs.CLcited by 143

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

Diffusion Language Models (DLMs) have been seen as a promising competitor for autoregressive language models. However, diffusion language models have long been constrained by slow inference. A core challenge is that their non-autoregressive architecture and bidirectional attention preclude the key-value cache that accelerates decoding. We address this bottleneck by proposing a KV-cache-like mechanism, delayed KV-Cache, for the denoising process of DLMs. Our approach is motivated by the observation that different tokens have distinct representation dynamics throughout the diffusion process. Accordingly, we propose a delayed and conditioned caching strategy for key and value states. We design two complementary variants to cache key and value step-by-step: (1) dKV-Cache-Decode, which provides almost lossless acceleration, and even improves performance on long sequences, suggesting that existing DLMs may under-utilise contextual information during inference. (2) dKV-Cache-Greedy, which has aggressive caching with reduced lifespan, achieving higher speed-ups with quadratic time complexity at the cost of some performance degradation. dKV-Cache, in final, achieves from 2-10x speedup in inference, largely narrowing the gap between ARs and DLMs. We evaluate our dKV-Cache on several benchmarks, delivering acceleration across general language understanding, mathematical, and code-generation benchmarks. Experiments demonstrate that cache can also be used in DLMs, even in a training-free manner from current DLMs.

Key facts. Cited by 143 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 52 of its 57 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

52 of 57 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  2. 2025Implicit Search via Discrete Diffusion: A Study on Chess27 Feb
  3. 2025Large Language Diffusion Models14 Feb
  4. 2025Theoretical Benefit and Limitation of Diffusion Language Model13 Feb
  5. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions10 Feb
  6. 2024Energy-Based Diffusion Language Models for Text Generation28 Oct
  7. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  8. 2024Promises, Outlooks and Challenges of Diffusion Language Modeling17 Jun
  9. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  10. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  11. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation16 May
  12. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises20 Feb
2 more
  1. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  2. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct

Other references · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022Scalable Diffusion Models with Transformerscited by 6,054
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
26 more
  1. 2023Let's Verify Step by Stepcited by 3,789
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  6. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  9. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  11. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  12. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  13. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  14. 2017Non-Autoregressive Neural Machine Translationcited by 463
  15. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  16. 2023DeepCache: Accelerating Diffusion Models for Freecited by 359
  17. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  18. 2024Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformerscited by 175
  19. 2023Cache Me if You Can: Accelerating Diffusion Models through Block Cachingcited by 151
  20. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 149
  21. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Cachingcited by 142
  22. 2024FasterCache: Training-Free Video Diffusion Model Acceleration with High Qualitycited by 122
  23. 2024DiTFastAttn: Attention Compression for Diffusion Transformer Modelscited by 115
  24. 2024Faster Diffusion via Temporal Attention Decompositioncited by 54
  25. 2024Pard: Permutation-Invariant Autoregressive Diffusion for Graph Generationcited by 25
  26. 2024Hash3D: Training-free Acceleration for 3D Generationcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 63; 2026: 80.

143 papers.

Sort:

By area:

  1. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  2. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
  3. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcingcited by 116
  4. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 88
  5. 2025A Survey on Latent Reasoningcited by 81
  6. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  7. 2025A Survey on Diffusion Language Modelscited by 78
  8. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  9. 2025Diffusion Language Models Know the Answer Before Decodingcited by 70
  10. 2025Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Evictioncited by 65
  11. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 65
  12. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
38 more
  1. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 57
  2. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 55
  3. 2025Any-Order Flexible Length Masked Diffusioncited by 51
  4. 2025Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principlescited by 45
  5. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 41
  6. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 40
  7. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 37
  8. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 34
  9. 2025Plan for Speed: Dilated Scheduling for Masked Diffusion Language Modelscited by 34
  10. 2025Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Modelscited by 34
  11. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 34
  12. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 33
  13. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  14. 2025AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Sizecited by 31
  15. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  16. 2025d2Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Cachingcited by 25
  17. 2025From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMscited by 24
  18. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 24
  19. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 23
  20. 2025Set Block Decoding is a Language Model Inference Acceleratorcited by 23
  21. 2025Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphscited by 21
  22. 2025Accelerating Diffusion LLM Inference via Local Determinism Propagationcited by 21
  23. 2025CDLM: Consistency Diffusion Language Models For Faster Samplingcited by 21
  24. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 20
  25. 2025Sequential Diffusion Language Modelscited by 19
  26. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  27. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 18
  28. 2025LoPA: Scaling dLLM Inference via Lookahead Parallel Decodingcited by 17
  29. 2025Inpainting-Guided Policy Optimization for Diffusion Large Language Modelscited by 16
  30. 2026dLLM: Simple Diffusion Language Modelingcited by 14
  31. 2026Improved Large Language Diffusion Modelscited by 14
  32. 2025Thinking Inside the Mask: In-Place Prompting in Diffusion LLMscited by 13
  33. 2025Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Modelscited by 13
  34. 2025UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Modelscited by 13
  35. 2025Parallel Sampling from Masked Diffusion Models via Conditional Independence Testingcited by 13
  36. 2025SDAR-VL: Stable and Efficient Block-wise Diffusion for Vision-Language Understandingcited by 13
  37. 2026DAWN: Dependency-Aware Fast Inference for Diffusion LLMscited by 13
  38. 2025Quant-dLLM: Post-Training Extreme Low-Bit Quantization for Diffusion Large Language Modelscited by 12

50 most cited of 143 All 143 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding
  2. 2026Archer: Adaptive Reuse of Cached Hidden States for Efficient Rollback in Diffusion Language Models
  3. 2026Energy Consumption Analysis of Discrete Diffusion and Autoregressive Language Models
  4. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
  5. 2026Diffusion Language Models: An Experimental Analysis
  6. 2026WaveFilter: Enhancing the Long-Context Capability of Diffusion LLMs via Wavelet-Guided KV Cache Filtering
  7. 2025Dream 7B: Diffusion Large Language Models
  8. 2025dInfer: An Efficient Inference Framework for Diffusion Language Models