mlsanityData to 7 Oct 2026

Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading

arXiv 2601.19910 ↗First posted 16 Dec 2025cs.AR cs.DCcited by 1

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

KV cache offloading enables long-context LLM inference by storing caches in CPU DRAM, but PCIe bandwidth limitations create severe bottlenecks. In this paper, we develops an analytical framework that derives kappacrit, the critical cached-to-prefill token ratio where execution becomes memory-bound and show typical workloads exceed this threshold by orders of magnitude. Empirical characterization reveals 99% of latency spent on transfers and serving offloaded requests results in GPU's consuming only 28% of their rated TDP, motivating our proposed optimizations for hardware interconnects, model architectures, and scheduling algorithms.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 27 references that are in the map. First posted on arXiv 16 Dec 2025.

Builds on

15 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 14

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  5. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  7. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  8. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  10. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  11. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 99
  12. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 88
2 more
  1. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputationcited by 12
  2. 2025MemAscend: System Memory Optimization for SSD-Offloaded LLM Fine-Tuningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  2. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
  3. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  4. 2026The KV Cache Is the New Memory Wall
  5. 2026PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding
  6. 2025Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
  7. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  8. 2025Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching