mlsanityData to 14 Oct 2026

LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences

arXiv 2510.11292 ↗First posted 13 Oct 2025cs.LG cs.AIcited by 4

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

While Key-Value (KV) cache succeeds in reducing redundant computations in auto-regressive models, it introduces significant memory overhead, limiting its practical deployment in long-sequence scenarios. Existing KV retrieval methods mitigate this by dynamically retaining only a subset of KV entries on the GPU. However, they still suffer from notable efficiency and accuracy bottlenecks due to per-token retrieval and coarse-grained page-level KV management, especially in long-output reasoning scenarios. With the emergence of large reasoning models, efficiently handling such scenarios has become increasingly important. To address this issue, we present two key observations: (1) critical KVs exhibit strong temporal locality during decoding, and (2) these KVs exhibit distinct distribution patterns across the input prompt and generated output. Building on these observations, we propose LouisKV, an efficient KV cache retrieval framework designed for various long-sequence scenarios. Specifically, LouisKV introduces a semantic-aware retrieval strategy leveraging temporal locality to trigger retrieval only at semantic boundaries, drastically reducing computation and data transfer overhead. LouisKV also designs a decoupled, fine-grained management scheme that tailors differentiated strategies for input and output sequences to create retrieval units that better match the model's attention patterns, enabling precise identification of critical KVs. Furthermore, to boost efficiency, LouisKV incorporates several kernel-level optimizations, including custom Triton and CUDA kernels to accelerate the KV clustering and retrieval. Evaluations show that LouisKV achieves up to 4.7× speedup over state-of-the-art KV retrieval methods while maintaining near-lossless accuracy across diverse long-sequence tasks, including long-input short-output, short-input long-output, and long-input long-output scenarios.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 32 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

29 of 32 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference19 May
  2. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  3. 2024RefreshKV: Updating Small KV Cache During Long-form Generation8 Nov
  4. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  6. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  7. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  10. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  11. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  12. 2019Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networkscited by 242
6 more
  1. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  2. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  3. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  4. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  5. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 50
  6. 2025RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoningcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025NOSA: Native and Offloadable Sparse Attentioncited by 4
  2. 2026Predict, Reuse, and Repair: Accelerating Dynamic Sparse Attention for Long-Context LLM Decodingcited by 2
  3. 2026Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stabilitycited by 1
  4. 2026From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs
  2. 2026SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
  3. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
  4. 2025LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Important
  5. 2026OVAL: Output-Aware Local Page Bases for KV Cache Retrieval
  6. 2025SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching
  7. 2026EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
  8. 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference