mlsanityData to 14 Oct 2026

CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference

arXiv 2604.08584 ↗First posted 30 Mar 2026cs.LG cs.AIcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context LLMs increasingly rely on extended, reusable prefill prompts for agents and domain Q&A, pushing attention and KV-cache to become the dominant decode-time bottlenecks. While sparse attention reduces computation and transfer costs, it often struggles to maintain accuracy at high sparsity levels due to the inherent distribution shift between Queries and Keys. We propose Centroid-Scoring Attention (CSAttention), a training-free sparse attention method optimized for high-throughput serving of reusable contexts. CSAttention adopts a storage-for-computation strategy tailored to the offline-prefill/online-decode setting: it front-loads computation into a one-time offline prefill phase that can be amortized across multiple queries, while aggressively optimizing per-step decoding latency. Specifically, CSAttention constructs query-centric lookup tables during offline prefill, whose size remains fixed during decoding, and enables online decoding to replace full-context scans with efficient table lookups and GPU-friendly score accumulation. Extensive experiments demonstrate that CSAttention achieves near-identical accuracy to full attention. Under high sparsity (95%) and long-context settings (32K-128K), CSAttention consistently outperforms state-of-the-art sparse attention methods in both model accuracy and inference speed, achieving up to 4.6x inference speedup over the most accurate baseline at a context length of 128K.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 29 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

22 of 29 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs16 Apr
  2. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  3. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  4. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text10 Oct
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  6. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  7. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  4. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  5. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  6. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  7. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  8. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  9. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  10. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  11. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  12. 2024HashAttention: Semantic Sparsity for Faster Inferencecited by 24
1 more
  1. 2025RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoningcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Context Memorization for Efficient Long Context Generationcited by 0
  2. 2026SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
  2. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
  3. 2026CLAA: Cross-Layer Attention Aggregation for Accelerating LLM Prefill
  4. 2026SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
  5. 2026You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
  6. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
  7. 2026TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration
  8. 2026Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference