mlsanityData to 14 Oct 2026

Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

arXiv 2606.02964 ↗First posted 1 Jun 2026cs.AR cs.CL cs.LGcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation. While approximate KV cache retention techniques reduce memory usage by sacrificing model accuracy, lossless approaches instead evict KV cache blocks from GPU memory and reconstruct them on demand to preserve exact outputs. Existing lossless KV cache management systems primarily base eviction decisions on access frequency or positional heuristics, without considering how different KV cache blocks affect the execution efficiency of GPU attention kernels. In this paper, we propose AsymCache, a computation-latency-aware KV cache management system for LLM inference that explicitly aligns cache residency decisions with GPU attention kernel performance, including three key components: Multi-Segment Attention (MSA) for efficient non-contiguous KV context processing, a cache eviction policy that jointly optimizes hit rate and position-aware recomputation cost, and an adaptive chunking scheduler for high hardware utilization. Experiments show that AsymCache reduces TTFT by up to 1.90-2.03x and time-per-output-token (TPOT) by 1.62-1.71x over latest baselines, confirming the effectiveness of the method in common workloads and validating its design goal of balancing computational efficiency with cache hit rate. Moreover, the low-level design of AsymCache allows seamless integration into agent serving systems such as Continuum, where it further reduces average job latency by up to 18.1%.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 56 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

30 of 56 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider3 Jun
  2. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference5 May
  3. 2024Fast State Restoration in LLM Serving with HCache7 Oct
  4. 2024Compute Or Load KV Cache? Why Not Both?4 Oct
  5. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  6. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  7. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  8. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  8. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  9. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  10. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  11. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  12. 2025SagaLLM: Context Management, Validation, and Transaction Guarantees for Multi-Agent LLM Planningcited by 60
9 more
  1. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 58
  2. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 44
  3. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Modelscited by 23
  4. 2024λ-Tune: Harnessing Large Language Models for Automated Database System Tuningcited by 19
  5. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  6. 2025Can Large Language Models Be Query Optimizer for Relational Databases?cited by 11
  7. 2025LEAP: LLM-powered End-to-end Automatic Library for Processing Social Science Queries on Unstructured Datacited by 7
  8. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Servingcited by 6
  9. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfigurationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026SARA: SLO-Aware Resource Allocation for Disaggregated Agentic LLM Servicescited by 0
  2. 2026When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reusecited by 0
  3. 2026Accelerating Cross-Model LLM Inference via Attention Map Reusecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  2. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Management
  4. 2026Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
  5. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
  7. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  8. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference