# Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction

arXiv: https://arxiv.org/abs/2508.02558 · HTML: https://mlsanity.com/paper/2508.02558/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- Efficient LLM inference and serving / [Diffusion LLM and MoE inference](https://mlsanity.com/inference/diffusion-llm-and-moe-inference/)

## Builds on (most cited first, 17 of 17)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding](https://mlsanity.com/paper/2505.22618/) (2025)
- [LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models](https://mlsanity.com/paper/2505.19223/) (2025)
- [MMaDA: Multimodal Large Diffusion Language Models](https://mlsanity.com/paper/2505.15809/) (2025)
- [dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching](https://mlsanity.com/paper/2506.06295/) (2025)
- [LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning](https://mlsanity.com/paper/2505.16933/) (2025)
- [dKV-Cache: The Cache for Diffusion Language Models](https://mlsanity.com/paper/2505.15781/) (2025)

## Built on by (most cited first, 30 of 65)

- [Fast-dLLM v2: Efficient Block-Diffusion LLM](https://mlsanity.com/paper/2509.26328/) (2025)
- [Diffusion Language Models Know the Answer Before Decoding](https://mlsanity.com/paper/2508.19982/) (2025)
- [DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding](https://mlsanity.com/paper/2510.02358/) (2025)
- [DPad: Efficient Diffusion Language Models with Suffix Dropout](https://mlsanity.com/paper/2508.14148/) (2025)
- [AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Size](https://mlsanity.com/paper/2509.26432/) (2025)
- [Accelerating Diffusion LLM Inference via Local Determinism Propagation](https://mlsanity.com/paper/2510.07081/) (2025)
- [ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding](https://mlsanity.com/paper/2512.13586/) (2025)
- [DMax: Aggressive Parallel Decoding for dLLMs](https://mlsanity.com/paper/2604.08302/) (2026)
- [DAWN: Dependency-Aware Fast Inference for Diffusion LLMs](https://mlsanity.com/paper/2602.06953/) (2026)
- [Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards](https://mlsanity.com/paper/2510.01544/) (2025)
- [How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices](https://mlsanity.com/paper/2510.18480/) (2025)
- [DiRL: An Efficient Post-Training Framework for Diffusion Language Models](https://mlsanity.com/paper/2512.22234/) (2025)
- [Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief States](https://mlsanity.com/paper/2510.11052/) (2025)
- [Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models](https://mlsanity.com/paper/2512.02044/) (2025)
- [TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Acceleration](https://mlsanity.com/paper/2602.08404/) (2026)
- [Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding](https://mlsanity.com/paper/2601.17917/) (2026)
- [Lookahead-then-Verify: Reliable Constrained Decoding for Diffusion LLMs under Context-Free Grammars](https://mlsanity.com/paper/2602.00612/) (2026)
- [Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing](https://mlsanity.com/paper/2602.02159/) (2026)
- [Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models](https://mlsanity.com/paper/2512.14008/) (2025)
- [Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Models](https://mlsanity.com/paper/2601.07351/) (2026)
- [Stop the Flip-Flop: Context-Preserving Verification for Fast Revocable Diffusion Decoding](https://mlsanity.com/paper/2602.06161/) (2026)
- [ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping](https://mlsanity.com/paper/2603.10088/) (2026)
- [Taming the Memory Footprint Crisis: System Design for Production Diffusion LLM Serving](https://mlsanity.com/paper/2512.17077/) (2025)
- [Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLM](https://mlsanity.com/paper/2606.26120/) (2026)
- [Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation](https://mlsanity.com/paper/2607.14557/) (2026)
- [One Token Is Enough: Improving Diffusion Language Models with a Sink Token](https://mlsanity.com/paper/2601.19657/) (2026)
- [Improving Variable-Length Generation in Diffusion Language Models via Length Regularization](https://mlsanity.com/paper/2602.07546/) (2026)
- [Rejection Mixing: Fast Semantic Propagation of Mask Tokens for Efficient DLLM Inference](https://mlsanity.com/paper/2602.22868/) (2026)
- [Breaking Block Boundaries: Anchor-based History-stable Decoding for Diffusion Large Language Models](https://mlsanity.com/paper/2604.08964/) (2026)
- [A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models](https://mlsanity.com/paper/2511.15098/) (2025)

## Abstract (arXiv)

Diffusion Large Language Models (dLLMs) enable breakthroughs in reasoning and parallel decoding but suffer from prohibitive quadratic computational complexity and memory overhead during inference. Current caching techniques accelerate decoding by storing full-layer states, yet impose substantial memory usage that limit long-context applications. Our analysis of attention patterns in dLLMs reveals persistent cross-layer sparsity, with pivotal tokens remaining salient across decoding steps and low-relevance tokens staying unimportant, motivating selective cache eviction. We propose Sparse-dLLM, the first training-free framework integrating dynamic cache eviction with sparse attention via delayed bidirectional sparse caching. By leveraging the stability of token saliency over steps, it retains critical tokens and dynamically evicts unimportant prefix/suffix entries using an attention-guided strategy. Extensive experiments on LLaDA and Dream series demonstrate Sparse-dLLM achieves up to 10× higher throughput than vanilla dLLMs, with comparable performance and similar peak memory costs, outperforming previous methods in efficiency and effectiveness. The code is available at https://github.com/OpenMOSS/Sparse-dLLM.