mlsanityData to 14 Oct 2026

AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference

arXiv 2605.12110 ↗First posted 12 May 2026cs.DCcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

As large language models scale to longer contexts, loading the growing KV cache during attention computation becomes a critical bottleneck. Previous work has shown that attention computation is dominated by a small subset of tokens. This motivates block sparse attention methods that partition the KV cache into fixed-size blocks and selectively compute attention over those blocks exhibiting high importance. However, these methods assign a uniform block size across all attention heads, implicitly assuming homogeneous behavior throughout the model. Our analysis reveals that this assumption is flawed: attention heads exhibit widely varying sensitivity to block granularity, and uniformity leads to suboptimal accuracy. We present AB-Sparse, a training-free algorithm-system co-designed framework that improves accuracy while preserving throughput. AB-Sparse introduces lightweight adaptive block size allocation across attention heads to improve accuracy. To compensate for the additional memory overhead, it further employs lossless block centroid quantization. In addition, custom GPU kernels are developed to support efficient execution with variable block sizes. Evaluation results demonstrate that AB-Sparse achieves an accuracy improvement of up to 5.43% over existing block sparse attention baselines without throughput overhead.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 32 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

25 of 32 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference19 May
  2. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  3. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  4. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  11. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  12. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
5 more
  1. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  2. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  3. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  4. 2024RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generationcited by 70
  5. 2024CuriousLLM: Elevating Multi-Document Question Answering with LLM-Enhanced Knowledge Graph Reasoningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Block-Sparse Attention with Semantic-Geometric Decoupled Routingcited by 0
  2. 2026Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
  2. 2025Training-free Context-adaptive Attention for Efficient Long Context Modeling
  3. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
  4. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  5. 2025Rectified Sparse Attention
  6. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
  7. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs
  8. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs