mlsanityData to 14 Oct 2026

SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers

arXiv 2606.22874 ↗First posted 22 Jun 2026cs.LG cs.AIcited by 5

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long contexts have become standard in pretrained LLMs, yet they remain expensive to run: prefill compute grows quadratically with sequence length, and every decode step re-reads a key-value cache that grows linearly with it. Sparse attention cuts these costs by attending only to a relevant subset of past tokens, but selecting that subset is itself expensive. We present SpotAttention, a lightweight selector that attaches to a frozen pretrained transformer and learns by KL distillation to estimate its attention distribution. The selector picks the top-K keys each query attends to, and because its estimate is a calibrated distribution, a dual top-p rule reads the per-query, per-layer budget directly from it. Across Qwen3 (dense, 4B-32B) and Qwen3.5 (hybrid linear/full attention, 4B-9B), SpotAttention matches dense accuracy at contexts up to 128K tokens, eight times the training length. Decode at L=128K runs 3.9x faster than FlashAttention and 1.8x faster than Twilight, the strongest training-free baseline. Quantizing the selector's K-cache to INT4 or FP4 microscale shrinks it 3.5x at no accuracy cost.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 39 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

32 of 39 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs17 Feb
  2. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  3. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  4. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  11. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  12. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
12 more
  1. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  2. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  3. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  4. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  5. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  6. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  7. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  8. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  9. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  10. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
  11. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 29
  12. 2026MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inferencecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Language Models Can Control Their Own Attentioncited by 1
  2. 2026Learning What Matters: Supervising Global Context Pruning with Causal Evidence Setscited by 0
  3. 2026On-Demand Attention: Language Models Know When to Recallcited by 0
  4. 2026The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trendscited by 0
  5. 2026MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers
  2. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
  3. 2026You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
  4. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
  5. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
  6. 2025SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformers
  7. 2024A Training-free Sub-quadratic Cost Transformer Model Serving Framework With Hierarchically Pruned Attention
  8. 2026FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving