mlsanityData to 14 Oct 2026

Towards Efficient Key-Value Cache Management for Prefix Prefilling in LLM Inference

arXiv 2505.21919 ↗First posted 28 May 2025cs.ET cs.AI cs.DCcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The increasing adoption of large language models (LLMs) with extended context windows necessitates efficient Key-Value Cache (KVC) management to optimize inference performance. Inference workloads like Retrieval-Augmented Generation (RAG) and agents exhibit high cache reusability, making efficient caching critical to reducing redundancy and improving speed. We analyze real-world KVC access patterns using publicly available traces and evaluate commercial key-value stores like Redis and state-of-the-art RDMA-based systems (CHIME [1] and Sherman [2]) for KVC metadata management. Our work demonstrates the lack of tailored storage solution for KVC prefilling, underscores the need for an efficient distributed caching system with optimized metadata management for LLM workloads, and provides insights into designing improved KVC management systems for scalable, low-latency inference.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 6 of its 14 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

6 of 14 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression17 Dec
  2. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct

Other references · 4

  1. 2020Longformer: The Long-Document Transformercited by 3,413
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  3. 2024Pie: Pooling CPU Memory for LLM Inferencecited by 13
  4. 2021Sherman: A Write-Optimized Distributed B+Tree Index on Disaggregated Memorycited by 1

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2025DeepSieve: Information Sieving via LLM-as-a-Knowledge-Routercited by 13

Similar, not cited

Close in wording, no citation either way

  1. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  2. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
  3. 2025Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs
  4. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  5. 2026ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management
  6. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  7. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  8. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference