mlsanityData to 14 Oct 2026

Predict, Reuse, and Repair: Accelerating Dynamic Sparse Attention for Long-Context LLM Decoding

arXiv 2606.30389 ↗First posted 29 Jun 2026cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Dynamic sparse attention (DSA) accelerates long-context LLM decoding by attending to only the top-K KV blocks relevant to each query, but it introduces a serialized selection-to-attention dependency that emerges as a new latency bottleneck. We present PRR, a speculate-reuse-repair runtime that exploits temporal locality in DSA selections to predict likely blocks, speculate the attention over them while selection is in flight, and incrementally repair missed blocks once the true selected set is known. PRR uses a lightweight EMA-based predictor, a profiling-guided speculation budget that keeps speculative work off the critical path, and a FlashAttention-based repair kernel that folds missed blocks into the partial attention state using online-softmax statistics. Across long-context benchmarks and representative DSA methods, PRR reduces per-token decoding latency by up to 40% while preserving downstream task accuracy. Github: https://github.com/Tianyu9748/Incremental_FlashAttention

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 21 of its 30 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

21 of 30 references.

References · 21

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Let's Verify Step by Stepcited by 3,789
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  9. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  10. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 335
  11. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  12. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
9 more
  1. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  2. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 224
  3. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  4. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 159
  5. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  6. 2025FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Managementcited by 5
  7. 2025LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequencescited by 4
  8. 2025AsyncSpade: Efficient Test-Time Scaling with Asynchronous Sparse Decodingcited by 2
  9. 2026Dynamic Sparse Attention: Access Patterns and Architecturecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Recall Before You Rank: Similarity-Guided Top-K Reuse for Efficient Long-Context Attentioncited by 1
  2. 2026SPIN: Shadow Predictive Indexer for Sparse Attentioncited by 0