mlsanityData to 7 Oct 2026

DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference

arXiv 2604.24647 ↗First posted 27 Apr 2026cs.CL cs.AIcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context reasoning is a critical capability of large language models (LLMs), enabling applications such as long-document understanding, summarization, and code generation. However, efficient autoregressive inference relies on the key-value (KV) cache, whose memory footprint grows linearly with sequence length, leading to a major memory bottleneck. To mitigate this overhead, KV cache pruning methods discard cached tokens with low attention scores during inference. Most existing methods apply a uniform pruning ratio across layers, implicitly assuming that all layers contribute equally to overall model performance. We show that this assumption is suboptimal, as layers differ significantly in their sensitivity to pruning. We propose DepthKV, a layer-dependent pruning framework that allocates a fixed global KV budget across layers based on their sensitivity, rather than using a uniform allocation. Across multiple models and tasks, DepthKV consistently outperforms uniform pruning at the same global pruning ratio, demonstrating more effective utilization of the KV cache budget through layer-dependent allocation.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 41 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

30 of 41 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs2 Mar
  2. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  3. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  4. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  5. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters18 Jun
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  10. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  11. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 18

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  9. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  10. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  11. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  12. 2021Efficient Attentions for Long Document Summarizationcited by 342
6 more
  1. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  2. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  3. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  4. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  5. 2023Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural languagecited by 32
  6. 2026Do Chatbot LLMs Talk Too Much? The YapBench Benchmarkcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agentscited by 3
  2. 2026Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibilitycited by 1
  3. 2026Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
  2. 2026Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inference
  3. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  4. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  5. 2024ThinK: Thinner Key Cache by Query-Driven Pruning
  6. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  7. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
  8. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning