mlsanityData to 14 Oct 2026

Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs

arXiv 2502.12216 ↗First posted 17 Feb 2025cs.LG cs.AI cs.CLcited by 24

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context models are essential for many applications but face inefficiencies in loading large KV caches during decoding. Prior methods enforce fixed token budgets for sparse attention, assuming a set number of tokens can approximate full attention. However, these methods overlook variations in the importance of attention across heads, layers, and contexts. To address these limitations, we propose Tactic, a sparsity-adaptive and calibration-free sparse attention mechanism that dynamically selects tokens based on their cumulative attention scores rather than a fixed token budget. By setting a target fraction of total attention scores, Tactic ensures that token selection naturally adapts to variations in attention sparsity. To efficiently approximate this selection, Tactic leverages clustering-based sorting and distribution fitting, allowing it to accurately estimate token importance with minimal computational overhead. We show that Tactic outperforms existing sparse attention algorithms, achieving superior accuracy and up to 7.29x decode attention speedup. This improvement translates to an overall 1.58x end-to-end inference speedup, making Tactic a practical and effective solution for long-context LLM inference in accuracy-sensitive applications.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 26 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

20 of 26 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  2. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  3. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  4. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct

Other references · 14

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  11. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  12. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
2 more
  1. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  2. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 17.

24 papers.

Sort:

By area:

  1. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 106
  2. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 29
  3. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 22
  4. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 18
  5. 2025Multipole Attention for Efficient Long Context Reasoningcited by 14
  6. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  7. 2025Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decodingcited by 10
  8. 2025vAttention: Verified Sparse Attentioncited by 6
  9. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMscited by 5
  10. 2026SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformerscited by 5
  11. 2026SPLA: Block Sparse Plus Linear Attention for Long Context Modelingcited by 3
  12. 2026SOCKET: SOft Collision Kernel EsTimator for Sparse Attentioncited by 3
12 more
  1. 2026MAGE: All-[MASK] Block Already Knows Where to Look in Block Diffusion LLMcited by 2
  2. 2026FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Servingcited by 2
  3. 2025Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretrainingcited by 1
  4. 2026S3-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inferencecited by 1
  5. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cachecited by 1
  6. 2026LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decodingcited by 1
  7. 2026EntmaxKV: Support-Aware Decoding for Entmax Attentioncited by 0
  8. 2026GEM-KMeans: Memory-Efficient and Accurate Clustering on Massive Scale with GPU Optimizationcited by 0
  9. 2026HHR: Hierarchical Hash Retrieval for Efficient LLM Generationcited by 0
  10. 2026MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decodingcited by 0
  11. 2026Attention via Black-Box Vector Searchcited by 0
  12. 2026Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Valuescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  2. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
  3. 2026From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs
  4. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
  5. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
  6. 2026You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
  7. 2026SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
  8. 2025Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference