mlsanityData to 14 Oct 2026

Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving

arXiv 2604.26837 ↗First posted 29 Apr 2026cs.LGcited by 4

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context LLM serving is bottlenecked by the cost of attending over ever-growing KV caches. Dynamic sparse attention promises relief by accessing only a small, query-dependent subset of the KV state per decoding step and extending the KV storage to CPU memory. In practice, however, these algorithmic savings rarely translate into end-to-end system-level gains because sparse methods typically operate at different granularities and thus rely on ad hoc, per-algorithm implementations. At the same time, hierarchical KV storage introduces a new systems bottleneck: retrieving fine-grained, irregular KV subsets across the GPU-CPU boundary can easily erase the benefits of sparsity. We present SPIN, a sparse-attention-aware inference framework that co-designs the execution pipeline with hierarchical KV storage through three techniques: (1) a unified partition abstraction that maps different sparsity granularities onto a shared page-based KV substrate; (2) a locality-aware KV cache manager that dynamically sizes per-request HBM budgets and uses a GPU-friendly bucketed LRU policy to cut PCIe round-trips; and (3) a two-level hierarchical metadata layout sized to the active working set rather than the worst-case address space. Built on vLLM with three representative sparse attention algorithms, SPIN delivers 1.66-5.66x higher end-to-end throughput and 7-9x lower TTFT than vLLM, and reduces TPOT by up to 58% over the original sparse-attention implementations.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 51 of its 70 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

51 of 70 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  2. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneity24 Mar
  3. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention20 Feb
  4. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection5 Nov
  5. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  6. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  7. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  10. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  11. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 40

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  11. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  12. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
28 more
  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  6. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  7. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  8. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  9. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  10. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  11. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  12. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  13. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  14. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 183
  15. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  16. 2019Adaptively Sparse Transformerscited by 155
  17. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
  18. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  19. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
  20. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 116
  21. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  22. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  23. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  24. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  25. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  26. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  27. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
  28. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inferencecited by 0
  2. 2026The Epistemic Politics of AI Anthropomorphismcited by 0
  3. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetchingcited by 0
  4. 2026SparseEngine: Sparse-First Inference Enginecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
  2. 2026HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management
  3. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  4. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  5. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
  6. 2026SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inference
  7. 2026SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
  8. 2026SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL