mlsanityData to 14 Oct 2026

Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding

arXiv 2609.00097 ↗First posted 31 Aug 2026cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

The development of long-context Large Language Models (LLMs) is constrained by the memory bandwidth bottleneck and quadratic complexity of the attention mechanism during decoding. To overcome the inherent trade-offs between the memory overhead of metadata-based metrics and the computational inefficiency of adaptive selection strategies, we present Faster Flash Decoding (FFD), a novel hardware-algorithm co-design framework designed to break the memory wall in long-context decoding. FFD integrates the selector and computer into a fully fused kernel, replacing external metadata indices with content-aware scanning via low-bit quantization. Furthermore, we introduce the top-delta strategy, which dynamically filters blocks to achieve distribution-adaptive sparsity without global synchronization. Offering a training-free and plug-and-play solution, FFD also enables the reuse of scanning results for computation, achieving up to 11.6x kernel-level speedup and scaling to 256K context length, with 2.37x end-to-end throughput improvement. Empirical validation on RULER and LongBench confirms that FFD maintains model accuracy while delivering high-ratio sparsity, with code available at https://github.com/qluoluo/faster-flash-decoding

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 27 of its 27 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

27 of 27 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  12. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
15 more
  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  3. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  4. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  6. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  7. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  8. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  9. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  10. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  11. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  12. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 71
  13. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
  14. 2025Thus Spake Long-Context Large Language Modelcited by 12
  15. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefillingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Cachescited by 0
  2. 2026FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backwardcited by 0
  3. 2026HHR: Hierarchical Hash Retrieval for Efficient LLM Generationcited by 0
  4. 2026MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding
  2. 2026Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding
  3. 2026SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inference
  4. 2026LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding
  5. 2026SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
  6. 2026ART: Attention Run-time Termination for Efficient Large Language Model Decoding
  7. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
  8. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs