mlsanityData to 14 Oct 2026

LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management

arXiv 2410.00428 ↗First posted 1 Oct 2024cs.DC cs.AI cs.LGcited by 16

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The expanding context windows in large language models (LLMs) have greatly enhanced their capabilities in various applications, but they also introduce significant challenges in maintaining low latency, particularly in Time to First Token (TTFT). This paper identifies that the sharp rise in TTFT as context length increases is predominantly driven by queuing delays, which are caused by the growing demands for GPU Key-Value (KV) cache allocation clashing with the limited availability of KV cache blocks. To address this issue, we propose LayerKV, a simple yet effective plug-in method that effectively reduces TTFT without requiring additional hardware or compromising output performance, while seamlessly integrating with existing parallelism strategies and scheduling techniques. Specifically, LayerKV introduces layer-wise KV block allocation, management, and offloading for fine-grained control over system memory, coupled with an SLO-aware scheduler to optimize overall Service Level Objectives (SLOs). Comprehensive evaluations on representative models, ranging from 7B to 70B parameters, across various GPU configurations, demonstrate that LayerKV improves TTFT latency up to 69x and reduces SLO violation rates by 28.7%, significantly enhancing the user experience.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 45 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

29 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving22 Jul
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  10. 2024Yi: Open Foundation Models by 01.AIcited by 719
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  12. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
12 more
  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  3. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  4. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  5. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  6. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  7. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  8. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  9. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  10. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 66
  11. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 54
  12. 2020Revisiting Linformer with a modified self-attention with linear complexitycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 7.

16 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  2. 2025KVComm: Enabling Efficient LLM Communication through Selective KV Sharingcited by 18
  3. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  4. 2026PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cachecited by 13
  5. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloadingcited by 9
  6. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 6
  7. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfigurationcited by 6
  8. 2025DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Accelerationcited by 4
  9. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  10. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inferencecited by 3
  11. 2026Adaptive KV Cache Reuse for Fast Long-Context LLM Servingcited by 3
  12. 2025Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloadingcited by 2
4 more
  1. 2025MatKV: Trading Compute for Flash Storage in LLM Inferencecited by 2
  2. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 1
  3. 2025Symbiosis: Multi-Adapter Inference and Fine-Tuningcited by 0
  4. 2026Persistent Computational State: A Session-Centric Runtime for Generative World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Strata: Hierarchical Context Caching for Long Context Language Model Serving
  2. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management
  3. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  4. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  5. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
  6. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
  7. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines
  8. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits