mlsanityData to 7 Oct 2026

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

arXiv 2606.04511 ↗First posted 3 Jun 2026cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Sparse attention reduces compute and memory bandwidth for long-context LLM inference. However, two key challenges remain: (1) KV cache capacity still grows with sequence length, and offloading to CPU memory introduces a PCIe transfer bottleneck; (2) the sparse selection step itself retains O(T2) complexity and can dominate attention cost at long contexts. We propose SparDA, a decoupled sparse attention architecture that introduces a fourth per-layer projection, the Forecast, alongside Query, Key, and Value. The Forecast predicts the KV blocks needed by the next layer, enabling lookahead selection that overlaps CPU-to-GPU prefetch with current-layer execution. Because Forecast is decoupled from the attention query, our GQA implementation uses one Forecast head per GQA group, reducing selection overhead versus the original multi-head selector. SparDA adds <0.5% parameters and trains only the Forecast projections by matching the original selector's attention distribution. On two sparse-pretrained 8B models, SparDA matches or slightly improves accuracy and delivers up to 1.25× prefill speedup and 1.7× decode speedup over the sparse-attention offload baseline. By enabling larger feasible batch sizes on a single GPU, SparDA further reaches up to 5.3× higher decode throughput than the non-offload sparse baseline. Our source code is available at https://github.com/NVlabs/SparDA.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 30 of its 35 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

30 of 35 references.

References · 30

  1. 2023Let's Verify Step by Stepcited by 3,598
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  7. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  8. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  9. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  10. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  11. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  12. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
18 more
  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  3. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  4. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  5. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  6. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  7. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  8. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  9. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  10. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  11. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  12. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  13. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  14. 2025LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contextscited by 25
  15. 2026HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attentioncited by 10
  16. 2025Self-Taught Agentic Long Context Understandingcited by 7
  17. 2025SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Servingcited by 5
  18. 2025NOSA: Native and Offloadable Sparse Attentioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attentioncited by 1
  2. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetchingcited by 0
  3. 2026KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
  2. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  3. 2026You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
  4. 2026FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
  5. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
  6. 2026CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference
  7. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  8. 2026AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention