mlsanityData to 14 Oct 2026

Recency/Frequency Adaptive KV Caching for Large Language Model Serving

arXiv 2606.21238 ↗First posted 19 Jun 2026cs.DC cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Key-value (KV) caching is a powerful technique for accelerating large language model inference and generation. Inference workloads are large and diverse, which makes them difficult to cache effectively. Existing cache management strategies adopt the least-recently-used policy for evicting cache blocks. However, LRU leads to multiple unrelated workloads flushing each other's caches. To address this, we integrate adaptive caching that dynamically allocates cache space between recently and frequently occurring KV blocks. Evaluations show that it improves the KV cache hit rate by up to 10.8% and reduces time to first token by up to 12.6% over naive vLLM on synthetic document question answering workloads, and 2.1% and 2.0% respectively on real-world conversation workloads. The method generalizes well to batch inference and demonstrates clear interpretability while effectively accommodating diverse workloads.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 19 Jun 2026.

Builds on

21 of 29 references.

References · 21

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  6. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  8. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  9. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  10. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  11. 2023Efficient Large Language Models: A Surveycited by 186
  12. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
9 more
  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  2. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  3. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  4. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 87
  5. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  6. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  7. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 31
  8. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  9. 2025Locality-aware Fair Scheduling in LLM Servingcited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
  2. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Management
  4. 2026Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective
  5. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  6. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  7. 2025KVCrush: Key value cache size-reduction using similarity in head-behaviour
  8. 2024RefreshKV: Updating Small KV Cache During Long-form Generation