# SparseD: Sparse Attention for Diffusion Language Models

arXiv: https://arxiv.org/abs/2509.24014 · HTML: https://mlsanity.com/paper/2509.24014/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 9 of its 24 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 9 of 9)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Dream 7B: Diffusion Large Language Models](https://mlsanity.com/paper/2508.15487/) (2025)
- [LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models](https://mlsanity.com/paper/2505.19223/) (2025)
- [A Reparameterized Discrete Diffusion Model for Text Generation](https://mlsanity.com/paper/2302.05737/) (2023)

## Built on by (most cited first, 17 of 17)

- [DMax: Aggressive Parallel Decoding for dLLMs](https://mlsanity.com/paper/2604.08302/) (2026)
- [LiteAttention: A Temporal Sparse Attention for Diffusion Transformers](https://mlsanity.com/paper/2511.11062/) (2025)
- [MultiAnimate: Pose-Guided Image Animation Made Extensible](https://mlsanity.com/paper/2602.21581/) (2026)
- [Taming the Memory Footprint Crisis: System Design for Production Diffusion LLM Serving](https://mlsanity.com/paper/2512.17077/) (2025)
- [FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion](https://mlsanity.com/paper/2602.05305/) (2026)
- [DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention](https://mlsanity.com/paper/2603.08026/) (2026)
- [MAGE: All-[MASK] Block Already Knows Where to Look in Block Diffusion LLM](https://mlsanity.com/paper/2602.14209/) (2026)
- [LoSA: Locality Aware Sparse Attention for Block-Wise Diffusion Language Models](https://mlsanity.com/paper/2604.12056/) (2026)
- [Attention-Based Sampler for Diffusion Language Models](https://mlsanity.com/paper/2604.08564/) (2026)
- [Elastic-dLLM: Position Preserving Context Compression and Augmentation of Diffusion LLMs](https://mlsanity.com/paper/2605.18165/) (2026)
- [Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques](https://mlsanity.com/paper/2607.12829/) (2026)
- [Exploring Motion-Language Alignment for Text-driven Motion Generation](https://mlsanity.com/paper/2604.02973/) (2026)
- [PulseCol: Periodically Refreshed Column-Sparse Attention for Accelerating Diffusion Language Models](https://mlsanity.com/paper/2605.20813/) (2026)
- [HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval](https://mlsanity.com/paper/2606.21633/) (2026)
- [Accelerating Diffusion Language Models via Structured Suffix Modeling](https://mlsanity.com/paper/2608.23167/) (2026)
- [Mask-Guided KV Cache Eviction in Block Diffusion Language Models](https://mlsanity.com/paper/2610.06996/) (2026)
- [Prefilling-dLLM v2: Learned Chunk Routing for Long-Context Diffusion Language Models](https://mlsanity.com/paper/2610.17868/) (2026)

## Abstract (arXiv)

While diffusion language models (DLMs) offer a promising alternative to autoregressive models (ARs), existing open-source DLMs suffer from high inference latency. This bottleneck is mainly due to the attention's quadratic complexity with respect to context length in computing all query-key pairs. Intuitively, to reduce this complexity, a natural strategy is to restrict attention to sparse patterns that retain only the most relevant connections. Such approaches are well-established in ARs, where attention follows fixed and clearly defined sparse patterns. However, in DLMs, we observe distinct sparsity behaviors: (1) attention patterns vary across heads, (2) attention patterns in each head remain highly similar across denoising steps, and (3) early denoising steps are critical for generation. These findings render sparse attention methods designed for ARs largely incompatible with DLMs, as they fail to capture head-specific structures and risk degrading generation when applied in early denoising steps. To address these challenges, we propose SparseD, a novel sparse attention method for DLMs. Leveraging the observations, SparseD only requires pre-computing head-specific sparse patterns one time, and reuses them across all steps. This prevents recomputing sparse patterns at each denoising step. Meanwhile, SparseD uses full attention in the early steps, then switches to sparse attention later to maintain generation quality. Together, these establish SparseD as a practical and efficient solution for deploying DLMs in long-context applications. Experimental results demonstrate that SparseD achieves lossless acceleration, delivering up to 1.50× speedup over FlashAttention at a 64k context length with 1,024 denoising steps.