mlsanityData to 7 Oct 2026

HieraSparse: Hierarchical Semi-Structured Sparse KV Attention

arXiv 2604.16864 ↗First posted 18 Apr 2026cs.DC cs.ARcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The deployment of long-context Large Language Models (LLMs) poses significant challenges due to the intense computational cost of self-attention and the substantial memory overhead of the Key-Value Cache (KV Cache). In this paper, we introduce textitHieraSparse, a hierarchical KV Cache compression framework with acceleration kernels that leverage GPU sparse tensor cores to speed up semi-structured KV Cache attention for both the prefill and decode phases. With the hierarchical design, our method allows for a flexible quality-sparsity trade-off and successfully converts sparsity into efficiency. Compared to the state-of-the-art decode method that utilizes unstructured sparsity, textitHieraSparse achieves 1.2× KV compression ratio and 4.57× attention speedup at the same sparsity level. Furthermore, we extended the semi-structured KV Cache pruning to the prefill stage, which demonstrated up to 1.85× attention speedup at the highest sparsity. Lastly, we evaluate the generation quality of textitHieraSparse with a simple magnitude-based pruning method, and the results show that 1.34× prefill and 1.71× decode attention speedup can be achieved without significant quality drop. The codebase can be found at https://github.com/psl-ntu/HieraSparse.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 46 of its 54 references that are in the map. First posted on arXiv 18 Apr 2026.

Builds on

46 of 54 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  2. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference28 May
  3. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse21 Feb
  4. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs19 Dec
  5. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  6. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  7. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  8. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  10. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2025Qwen3 Technical Reportcited by 8,195
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2023Mistral 7Bcited by 3,065
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
22 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2024A Survey on Large Language Models for Code Generationcited by 646
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  8. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  9. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  10. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  11. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  12. 2021Accelerating Sparse Deep Neural Networkscited by 213
  13. 2024Many-Shot In-Context Learningcited by 203
  14. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  15. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  16. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  17. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  18. 2025TileLang: A Composable Tiled Programming Model for AI Systemscited by 42
  19. 2022Dynamic N:M Fine-grained Structured Sparse Attention Mechanismcited by 14
  20. 2023VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Corescited by 13
  21. 2024Enabling Unstructured Sparse Acceleration on Structured Sparse Acceleratorscited by 8
  22. 2025LeanK: Learnable K Cache Channel Pruning for Efficient Decodingcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  2. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  3. 2026Learning how to Forget: Fine-tuning for Long-Context Sparse Attention
  4. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
  5. 2026HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management
  6. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  7. 2025SALS: Sparse Attention in Latent Space for KV cache Compression
  8. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction