mlsanityData to 14 Oct 2026

ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse

arXiv 2605.22850 ↗First posted 16 May 2026cs.DC cs.AIcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Prefix KV caching has become a key mechanism in LLM serving: it reduces time to first token (TTFT) by avoiding redundant computation across requests that share a prefix (i.e., the system prompt). However, the accumulated KV cache is often larger than what GPU memory and local DRAM can hold. To preserve latency, current systems keep the KV cache in remote DRAM pools, increasing serving-cluster size and cost. In this paper, we explore a different approach: storing the KV cache in S3-compatible object storage so that capacity is no longer the constraint, while minimizing the impact on TTFT. We propose ObjectCache, which co-designs the storage protocol and transfer schedule so that the storage server delivers KV cache data in the order the GPU consumes it, overlapping data transfer with compute across concurrent requests. We prototype ObjectCache on a 100 Gbps RoCE cluster with NIXL (an inference library that abstracts storage and memory), Ceph RGW (an Object Gateway for clusters), and DAOS (an open source storage system). For 64K contexts, common in today's systems, ObjectCache adds only 5.6% latency over local DRAM; for 4K contexts, where less compute is available to mask transfer, ObjectCache adds 56--75,ms over the optimal local layerwise baseline. Under shared bandwidth caps, our scheduler reduces added TTFT by 1.2--1.8x compared with equal bandwidth sharing.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 70 references that are in the map. First posted on arXiv 16 May 2026.

Builds on

39 of 70 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter16 Apr
  2. 2026ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management20 Jan
  3. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  4. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider3 Jun
  5. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  7. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  10. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  11. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
2 more
  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  12. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
13 more
  1. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  2. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  3. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  4. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 87
  5. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  6. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  7. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
  8. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 32
  9. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  10. 2025DeepServe: Serverless Large Language Model Serving at Scalecited by 18
  11. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 11
  12. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 6
  13. 2025On 10x Better Scalability: KV Stores Scale Up KV Cachecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
  2. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
  3. 2025AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
  4. 2026TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash
  5. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  6. 2026Unified KV Pooling to Accelerate Long-Context LLM Serving
  7. 2026Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service
  8. 2026A CXL Memory Rack for Multi-Turn LLM Serving