mlsanityData to 7 Oct 2026

Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs

arXiv 2602.05191 ↗First posted 5 Feb 2026cs.LG cs.AIcited by 5

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

As long-context inference becomes central to large language models (LLMs), attention over growing key-value caches emerges as a dominant decoding bottleneck, motivating sparse attention for scalable inference. Fixed-budget top-k sparse attention cannot adapt to heterogeneous attention distributions across heads and layers, whereas top-p sparse attention directly preserves attention mass and provides stronger accuracy guarantees. Existing top-p methods, however, fail to jointly optimize top-p accuracy, selection overhead, and sparse attention cost, which limits their overall efficiency. We present Double-P, a hierarchical sparse attention framework that optimizes all three stages. Double-P first performs coarse-grained top-p estimation at the cluster level using size-weighted centroids, then adaptively refines computation through a second top-p stage that allocates token-level attention only when needed. Across long-context benchmarks, Double-P consistently achieves near-zero accuracy drop, reducing attention computation overhead by up to 1.8x and delivers up to 1.3x end-to-end decoding speedup over state-of-the-art fixed-budget sparse attention methods.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 41 of its 47 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

41 of 47 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention20 Feb
  2. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs17 Feb
  3. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  4. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  5. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  6. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  11. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
18 more
  1. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  2. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  3. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  4. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  5. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  6. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  7. 2025Qwen2.5-1M Technical Reportcited by 158
  8. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  9. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  10. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  11. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  12. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  13. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 94
  14. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  15. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 78
  16. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  17. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  18. 2025VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inferencecited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attentioncited by 10
  2. 2026DashAttention: Differentiable and Adaptive Sparse Hierarchical Attentioncited by 3
  3. 2026Block Sparse Attention with Log-Linear Complexitycited by 0
  4. 2026MassAlloc Attention: Let Attention Allocate Its Own Computecited by 0
  5. 2026MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
  2. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  3. 2025Training-free Context-adaptive Attention for Efficient Long Context Modeling
  4. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
  5. 2026UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training
  6. 2026LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding
  7. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
  8. 2026You Only Index Once: Cross-Layer Sparse Attention with Shared Routing