Adaptive KV Cache Reuse for Fast Long-Context LLM Serving
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
In long-context Large Language Model (LLM) inference, the Time-To-First-Token (TTFT) latency incurred by the prefill stage has become the foremost bottleneck limiting interactive performance and deployment cost. KV Cache reuse offers a direct path to reduce redundant prefill, yet traditional prefix caching applies only to strict-prefix scenarios; directly reusing KV Cache in non-prefix settings breaks the cross-chunk global attention relationships and causes significant degradation in generation quality. When reusable KV Cache is offloaded to GPU-external cache pools, I/O overheads across heterogeneous hardware tiers further emerge as a new TTFT bottleneck. Efficient non-prefix KV Cache reuse therefore requires both semantic-consistency recovery and compute-I/O co-optimization. This paper presents CacheTune, a frequency-guided and hardware-aware KV Cache reuse system for long-context LLM serving. CacheTune first identifies, offline, the KV pairs most critical to cross-attention recovery through frequency-domain analysis, and then selectively recomputes only these semantic-critical tokens online while reusing the remaining KVs. To turn this semantic selection into end-to-end latency reduction, CacheTune further combines sparse KV transfer, multi-stream asynchronous overlap, deferred positional-encoding recovery, and hardware-aware adaptive recomputation-ratio tuning to balance computation and data movement across heterogeneous cache pools. Evaluations on mainstream LLMs and long-context tasks show that CacheTune achieves 3.72x-4.86x TTFT speedup and 3.93x-6.21x higher throughput while maintaining generation quality close to full recompute. Even when caches are offloaded to I/O-bound SSD/HDD storage, CacheTune sustains 2.34x-2.36x TTFT speedup through adaptive recomputation.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 37 of its 49 references that are in the map. First posted on arXiv 20 May 2026.
Builds on
37 of 49 references.
Closest ancestors · 15
Same direction, newest first
- 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference18 May
- 2026ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation31 Jan
- 2025CacheClip: Accelerating RAG with Effective KV Cache Reuse11 Oct
- 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
- 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference5 May
- 2025FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension1 May
- 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse21 Feb
- 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation5 Feb
- 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management1 Oct
- 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
- 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
- 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
3 more
Other references · 22
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
- 2023Mistral 7Bcited by 3,065
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
- 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
- 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
10 more
- 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
- 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
- 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
- 2020Learning in the Frequency Domaincited by 196
- 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
- 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
- 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 45
- 2023Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operatorcited by 19
- 2023An Introduction to the Compute Express Link (CXL) Interconnectcited by 9
- 2022DIVISION: Memory Efficient Training via Dual Activation Precisioncited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
- 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse
- 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
- 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
- 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
- 2025Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA
- 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
- 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference