mlsanityData to 7 Oct 2026

Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention

arXiv 2403.19708 ↗First posted 23 Mar 2024cs.CL cs.LGcited by 159

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Interacting with humans through multi-turn conversations is a fundamental feature of large language models (LLMs). However, existing LLM serving engines executing multi-turn conversations are inefficient due to the need to repeatedly compute the key-value (KV) caches of historical tokens, incurring high serving costs. To address the problem, this paper proposes CachedAttention, a new attention mechanism that enables reuse of KV caches across multi-turn conversations, significantly reducing the repetitive computation overheads. CachedAttention maintains a hierarchical KV caching system that leverages cost-effective memory/storage mediums to save KV caches for all requests. To reduce KV cache access overheads from slow mediums, CachedAttention employs layer-wise pre-loading and asynchronous saving schemes to overlap the KV cache access with the GPU computation. To ensure that the KV caches to be accessed are placed in the fastest hierarchy, CachedAttention employs scheduler-aware fetching and eviction schemes to consciously place the KV caches in different layers based on the hints from the inference job scheduler. To avoid the invalidation of the saved KV caches incurred by context window overflow, CachedAttention enables the saved KV caches to remain valid via decoupling the positional encoding and effectively truncating the KV caches. Extensive experimental results demonstrate that CachedAttention significantly decreases the time to the first token (TTFT) by up to 87%, improves the prompt prefilling throughput by up to 7.8× for multi-turn conversations, and reduces the end-to-end inference cost by up to 70%.

Key facts. Cited by 159 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 41 of its 56 references that are in the map. First posted on arXiv 23 Mar 2024.

Builds on

41 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  2. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  3. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  5. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023Mistral 7Bcited by 3,065
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
24 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  4. 2024Mixtral of Expertscited by 1,785
  5. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  6. 2021Transformer in Transformercited by 632
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  9. 2022Efficiently Scaling Transformer Inferencecited by 529
  10. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  12. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  13. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  14. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  15. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  16. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  17. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  18. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  19. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  20. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  21. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  22. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  23. 2018FRAGE: Frequency-Agnostic Word Representationcited by 47
  24. 2018Fast and exact analysis for LRU cachescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 53; 2026: 87.

159 papers.

Sort:

By area:

  1. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  3. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  6. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  7. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  8. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  9. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  10. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 38
  11. 2024Fast State Restoration in LLM Serving with HCachecited by 32
  12. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 31
38 more
  1. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  2. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  3. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  4. 2024Compute Or Load KV Cache? Why Not Both?cited by 25
  5. 2024SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactionscited by 25
  6. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 25
  7. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 24
  8. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  9. 2024Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Executioncited by 16
  10. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  11. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  12. 2025KVComm: Enabling Efficient LLM Communication through Selective KV Sharingcited by 16
  13. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  14. 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inferencecited by 15
  15. 2025Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregationcited by 14
  16. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 14
  17. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Servingcited by 13
  18. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  19. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 13
  20. 2025Optimizing Agentic Language Model Inference via Speculative Tool Callscited by 11
  21. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  22. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  23. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 10
  24. 2025SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Modelscited by 9
  25. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  26. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 8
  27. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharingcited by 8
  28. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Managementcited by 7
  29. 2024Generative Prompt Internalizationcited by 6
  30. 2025TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systemscited by 6
  31. 2025SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matchingcited by 6
  32. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfigurationcited by 6
  33. 2026ICaRus: Identical Cache Reuse for Efficient Multi Model Inferencecited by 6
  34. 2024DiskGNN: Bridging I/O Efficiency and Model Accuracy for Out-of-Core GNN Trainingcited by 5
  35. 2025CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIMcited by 5
  36. 2025DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Accelerationcited by 5
  37. 2025PiKV: KV Cache Management System for Mixture of Expertscited by 5
  38. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5

50 most cited of 159 All 159 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
  3. 2026Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving
  4. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  5. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  6. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  7. 2026Back from the Future: Key-Value Cache Management by Counter-Causal Surprise
  8. 2026Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM