mlsanityData to 7 Oct 2026

HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management

arXiv 2608.07009 ↗First posted 7 Aug 2026cs.DCcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Top-k sparse attention makes long-context LLM decoding cheap to compute: each step reads only a few thousand selected KV entries rather than the full context. Serving systems, however, typically keep the entire KV cache in GPU HBM so that every position stays selectable, so a request's memory bill still grows with its full context length--decoding hits a capacity wall long before it runs out of compute, and a context whose KV cache exceeds HBM cannot be served at all. We present HiSparse, an exact, indexer-agnostic hierarchical KV cache for sparse-attention serving. HiSparse keeps each request's full KV history in host memory and bounds its decode footprint with a small, fixed-size GPU cache; a fused CUDA kernel resolves each layer's selections--hit detection, LRU replacement, and host-to-device fetches--inside the decode CUDA graph; and, for models that share selections across layers, exact layer-wise prefetching hides roughly half of the remaining miss overhead. Because only KV placement changes, model outputs are unchanged. HiSparse is merged into upstream SGLang and evaluated across three sparse-attention families (DSA, NSA, and Quest) on H200, B200, and GH200 platforms: it improves peak generation throughput by up to 4.7x on long-context workloads while preserving comparable per-token latency and reducing time-to-first-token at high load--and a no-IO oracle shows the resolution mechanism itself adds no measurable per-token cost, leaving host-device IO as the only price of bounded residency.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 42 references that are in the map. First posted on arXiv 7 Aug 2026.

Builds on

29 of 42 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Strata: Hierarchical Context Caching for Long Context Language Model Serving26 Aug
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  4. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  6. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 21

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2020Longformer: The Long-Document Transformercited by 3,362
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2020Reformer: The Efficient Transformercited by 1,658
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  9. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
9 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  2. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  3. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  4. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  5. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  6. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  7. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  8. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  9. 2025ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Expcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Rethinking Heterogeneous System Disaggregation for Subquadratic Attentioncited by 0
  2. 2026SparseEngine: Sparse-First Inference Enginecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  2. 2026HieraSparse: Hierarchical Semi-Structured Sparse KV Attention
  3. 2025NOSA: Native and Offloadable Sparse Attention
  4. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
  5. 2026SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL
  6. 2025PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel
  7. 2026Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention
  8. 2026SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inference