mlsanityData to 14 Oct 2026

ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching

arXiv 2509.16857 ↗First posted 21 Sep 2025cs.DC cs.AI cs.LGcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Distributed prefix caching accelerates long-context LLM serving by reusing KV cache entries for common context prefixes. However, KV cache fetches can become a bottleneck when network bandwidth is limited. Compression mitigates the bandwidth issue, but can degrade overall performance when decompression interferes with model computation. We present ShadowServe, the first SmartNIC-accelerated, interference-free prefix caching system for LLM serving. ShadowServe separates a control plane on the host and a data plane fully offloaded to the SmartNIC, which eliminates interference to both host GPU and CPU. To overcome the SmartNIC's limited compute and memory resources, we design a chunked pipeline that parallelizes data plane operations across the SmartNIC's compute resources, and a minimal-copy memory management scheme that reduces memory pressure on the SmartNIC. Compared to state-of-the-art solutions, ShadowServe achieves up to 2.2x lower loaded time-per-output-token (TPOT), and reduces time-to-first-token (TTFT) by up to 1.38x in low-bandwidth scenarios (<= 20 Gbps), translating to up to 1.35x higher throughput.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 51 of its 111 references that are in the map. First posted on arXiv 21 Sep 2025.

Builds on

51 of 111 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider3 Jun
  2. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving18 Mar
  3. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse17 Mar
  4. 2024Compute Or Load KV Cache? Why Not Both?4 Oct
  5. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  7. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  8. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  9. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  10. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  11. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
1 more
  1. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 38

  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  5. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  9. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  10. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  11. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  12. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
26 more
  1. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  2. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  3. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  4. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90
  5. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 87
  6. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  7. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  8. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  9. 2025Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Modelscited by 62
  10. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  11. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  12. 2021tf.data: A Machine Learning Data Processing Frameworkcited by 41
  13. 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31
  14. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  15. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
  16. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 22
  17. 2024SkyServe: Serving AI Models across Regions and Clouds with Spot Instancescited by 20
  18. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  19. 2019Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applicationscited by 17
  20. 2022Skyplane: Optimizing Transfer Cost and Throughput Using Cloud-Aware Overlayscited by 5
  21. 2021FlexTOE: Flexible TCP Offload with Fine-Grained Parallelismcited by 4
  22. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharingcited by 4
  23. 2025The Streaming Batch Model for Efficient and Fault-Tolerant Heterogeneous Executioncited by 3
  24. 2022Characterizing Off-path SmartNIC for Accelerating Distributed Systemscited by 2
  25. 2019λ-NIC: Interactive Serverless Compute on Programmable SmartNICscited by 1
  26. 2024Demystifying Datapath Accelerator Enhanced Off-path SmartNICcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  2. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  3. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving
  2. 2026CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving
  3. 2026PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding
  4. 2026Sparse Prefix Caching for Hybrid and Recurrent LLM Serving
  5. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  6. 2026PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
  7. 2026MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression
  8. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading