mlsanityData to 14 Oct 2026

Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction

arXiv 2508.02558 ↗First posted 4 Aug 2025cs.CLcited by 65

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Diffusion Large Language Models (dLLMs) enable breakthroughs in reasoning and parallel decoding but suffer from prohibitive quadratic computational complexity and memory overhead during inference. Current caching techniques accelerate decoding by storing full-layer states, yet impose substantial memory usage that limit long-context applications. Our analysis of attention patterns in dLLMs reveals persistent cross-layer sparsity, with pivotal tokens remaining salient across decoding steps and low-relevance tokens staying unimportant, motivating selective cache eviction. We propose Sparse-dLLM, the first training-free framework integrating dynamic cache eviction with sparse attention via delayed bidirectional sparse caching. By leveraging the stability of token saliency over steps, it retains critical tokens and dynamically evicts unimportant prefix/suffix entries using an attention-guided strategy. Extensive experiments on LLaDA and Dream series demonstrate Sparse-dLLM achieves up to 10× higher throughput than vanilla dLLMs, with comparable performance and similar peak memory costs, outperforming previous methods in efficiency and effectiveness. The code is available at https://github.com/OpenMOSS/Sparse-dLLM.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 17 of its 21 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

17 of 21 references.

References · 17

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  11. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  12. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
5 more
  1. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  2. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  3. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  4. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  5. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 143

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 47.

65 papers.

Sort:

By area:

  1. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
  2. 2025Diffusion Language Models Know the Answer Before Decodingcited by 70
  3. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 42
  4. 2025DPad: Efficient Diffusion Language Models with Suffix Dropoutcited by 39
  5. 2025AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Sizecited by 31
  6. 2025Accelerating Diffusion LLM Inference via Local Determinism Propagationcited by 21
  7. 2025ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decodingcited by 18
  8. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 18
  9. 2026DAWN: Dependency-Aware Fast Inference for Diffusion LLMscited by 13
  10. 2025Advancing Reasoning in Diffusion Language Models with Denoising Process Rewardscited by 10
  11. 2025How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practicescited by 10
  12. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Modelscited by 10
38 more
  1. 2025Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief Statescited by 9
  2. 2025Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Modelscited by 9
  3. 2026TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Accelerationcited by 8
  4. 2026Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decodingcited by 7
  5. 2026Lookahead-then-Verify: Reliable Constrained Decoding for Diffusion LLMs under Context-Free Grammarscited by 7
  6. 2026Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusingcited by 7
  7. 2025Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Modelscited by 6
  8. 2026Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Modelscited by 6
  9. 2026Stop the Flip-Flop: Context-Preserving Verification for Fast Revocable Diffusion Decodingcited by 6
  10. 2026ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skippingcited by 6
  11. 2025Taming the Memory Footprint Crisis: System Design for Production Diffusion LLM Servingcited by 5
  12. 2026Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLMcited by 5
  13. 2026Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncationcited by 5
  14. 2026One Token Is Enough: Improving Diffusion Language Models with a Sink Tokencited by 4
  15. 2026Improving Variable-Length Generation in Diffusion Language Models via Length Regularizationcited by 4
  16. 2026Rejection Mixing: Fast Semantic Propagation of Mask Tokens for Efficient DLLM Inferencecited by 4
  17. 2026Breaking Block Boundaries: Anchor-based History-stable Decoding for Diffusion Large Language Modelscited by 4
  18. 2025A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Modelscited by 3
  19. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 3
  20. 2026FOCUS: DLLMs Know How to Tame Their Compute Boundcited by 3
  21. 2026Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Modelscited by 3
  22. 2025DiffuTester: Accelerating Unit Test Generation for Diffusion LLMs via Mining Structural Patterncited by 2
  23. 2026LoSA: Locality Aware Sparse Attention for Block-Wise Diffusion Language Modelscited by 2
  24. 2026GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inferencecited by 2
  25. 2026DIP: Dynamic In-Context Planner For Diffusion Language Modelscited by 1
  26. 2026SPA-Cache: Singular Proxies for Adaptive Caching in Diffusion Language Modelscited by 1
  27. 2026GeoBlock: Inferring Block Granularity from Dependency Geometry in Diffusion Language Modelscited by 1
  28. 2026STDec: Spatio-Temporal Stability Guided Decoding for dLLMscited by 1
  29. 2026DepCap: Adaptive Block-Wise Parallel Decoding for Efficient Diffusion LM Inferencecited by 1
  30. 2026WaveFilter: Enhancing the Long-Context Capability of Diffusion LLMs via Wavelet-Guided KV Cache Filteringcited by 1
  31. 2026Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Modelscited by 1
  32. 2026Follow the Latent Roadmap: Navigating Revocable Decoding for Diffusion LLMs with Anchor Tokenscited by 1
  33. 2026VoidPadding: Let [VOID] Handle Padding in Masked Diffusion Language Models so that [EOS] Can Focus on Semantic Terminationcited by 1
  34. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniquescited by 1
  35. 2026Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMscited by 1
  36. 2025STaRR: Spatial-Temporal Token-Dynamics-Aware Responsive Remasking for Diffusion Language Modelscited by 0
  37. 2026FourierSampler: Unlocking Non-Autoregressive Potential in Diffusion Language Models via Frequency-Guided Generationcited by 0
  38. 2026AR-MAP: Are Autoregressive Large Language Models Implicit Teachers for Diffusion Large Language Models?cited by 0

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Mask Tokens as Prophet: Fine-Grained Cache Eviction for Efficient dLLM Inference
  2. 2025d2Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching
  3. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference
  4. 2025Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Models
  5. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  6. 2026Sangam: Efficiently Serving Diffusion LLMs with the AR Stack
  7. 2026Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Sampling
  8. 2026LEAP: Unlocking dLLM Parallelism via Lookahead Early-Convergence Token Detection