mlsanityData to 7 Oct 2026

Compute Or Load KV Cache? Why Not Both?

arXiv 2410.03065 ↗First posted 4 Oct 2024cs.LGcited by 25

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models (LLMs) are increasingly deployed in large-scale online services, enabling sophisticated applications. However, the computational overhead of generating key-value (KV) caches in the prefill stage presents a major bottleneck, particularly for long-context inputs. Prefix caching mitigates this issue by storing KV caches for reuse, reducing redundant computation. Despite its advantages, prefix caching suffers from high latency due to the limited I/O bandwidth of storage devices, constraining inference efficiency. To address this challenge, we introduce Cake, a novel KV cache loading system that optimally utilizes both computational and I/O resources in parallel. Cake employs a bidirectional scheduling strategy that dynamically balances KV cache computation and loading, ensuring efficient resource utilization. Additionally, Cake incorporates an adaptive scheduling mechanism that seamlessly integrates with non-prefix caching requests, improving system throughput and adapting to fluctuating resource availabilty. Through extensive evaluations across various hardware configurations, datasets, and storage conditions, Cake achieves on average 2.6x reduction in Time to First Token (TTFT) compared to compute-only and I/O-only methods. Our findings highlight Cake as an effective and practical solution for optimizing long-context LLM inference, bridging the gap between computation and I/O efficiency in large-scale AI deployments.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 43 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

25 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  2. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  9. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  10. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  11. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  12. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
9 more
  1. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  2. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  3. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  4. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 104
  5. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  6. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  7. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  8. 2022Skyplane: Optimizing Transfer Cost and Throughput Using Cloud-Aware Overlayscited by 5
  9. 2024Plato: Plan to Efficiently Decode for Large Language Model Inferencecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 10.

25 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  3. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  4. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 25
  5. 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Taskscited by 18
  6. 2025KVComm: Enabling Efficient LLM Communication through Selective KV Sharingcited by 16
  7. 2025Compression Barriers for Autoregressive Transformerscited by 5
  8. 2026CoMem: Context Management with A Decoupled Long-Context Modelcited by 5
  9. 2026Leyline: KV Cache Directives for Agentic Inferencecited by 5
  10. 2026MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inferencecited by 5
  11. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  12. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
13 more
  1. 2025Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decodingcited by 2
  2. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Servingcited by 2
  3. 2025SubGCache: Accelerating Graph-based RAG with Subgraph-level KV Cachecited by 1
  4. 2025Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharingcited by 1
  5. 2025End-to-End Transformer Acceleration Through Processing-in-Memory Architecturescited by 1
  6. 2026SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inferencecited by 1
  7. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restorationcited by 1
  8. 2026A Token/KV-Cache Communication Media Selection and Resource Allocation Strategy for Multi-Agent Collaborationcited by 1
  9. 2026LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Servingcited by 1
  10. 2025Learn from the Past: Fast Sparse Indexing for Large Language Model Decodingcited by 0
  11. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
  12. 2025Unlocking the Address Book: Dissecting the Sparse Semantic Structure of LLM Key-Value Caches via Sparse Autoencoderscited by 0
  13. 2026TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Cachingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management
  2. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  3. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
  4. 2025AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
  5. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  6. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  7. 2026Adaptive KV Cache Reuse for Fast Long-Context LLM Serving
  8. 2026Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service