mlsanityData to 14 Oct 2026

SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharing

arXiv 2606.16135 ↗First posted 15 Jun 2026cs.DCcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Multi-turn conversation is a fundamental scenario in LLM applications, widely used in chatbots and AI agents. As the conversation evolves, historical tokens accumulate continuously. Existing systems cache their key-value (KV) pairs to avoid redundant computation. However, limited GPU memory (HBM) capacity often forces these KV caches to be offloaded to CPU memory or SSD, making KV cache reloads increasingly costly in terms of latency as the context grows. Meanwhile, the constrained HBM capacity also limits the maximum inference length, thereby restricting the number of turns that can be supported in a conversation. To address these two challenges, we propose SwiftCache, a collaborative inference system that enables heterogeneous models to share underutilized GPU memory and NVLink bandwidth within a server. Specifically, models with low KV cache demand donate idle GPU memory to store the prefix cache of high-demand models, allowing cross-model KV cache sharing over NVLink and avoiding slow PCIe transfers. SwiftCache further reduces memory pressure by keeping only the KV cache of the currently active layer in local GPU memory, thereby enabling longer-context inference. Our experiments on real-world workloads show that SwiftCache reduces P99 time-to-first-token (TTFT) by up to 69% and extends maximum context length by up to 3.98x compared to vLLM and SGLang, with minimal interference to co-located models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 53 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

29 of 53 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  2. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneity24 Mar
  3. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction4 Dec
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  6. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  7. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  8. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  9. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
7 more
  1. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  2. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  3. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  4. 2024DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Servingcited by 35
  5. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  6. 2025DeepServe: Serverless Large Language Model Serving at Scalecited by 18
  7. 2025BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026A Universal Context-Reuse Layer for Cross-Model KV Sharingcited by 2
  2. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  2. 2026Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference
  3. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
  4. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  5. 2026RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse
  6. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  7. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  8. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling