mlsanityData to 7 Oct 2026

ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty

arXiv 2412.09036 ↗First posted 12 Dec 2024cs.CLcited by 7

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language models (LLMs) have become a research hotspot. To accelerate the inference of LLMs, storing computed caches in memory has become the standard technique. However, as the inference length increases, growing KV caches might lead to out-of-memory issues. Many existing methods address this issue through KV cache compression, primarily by preserving key tokens throughout all layers to reduce information loss. Most of them allocate a uniform budget size for each layer to retain. However, we observe that the minimum budget sizes needed to retain essential information vary across layers and models based on the perspectives of attention and hidden state output. Building on this observation, this paper proposes a simple yet effective KV cache compression method that leverages layer uncertainty to allocate budget size for each layer. Experimental results show that the proposed method can reduce memory usage of the KV caches to only ~20% when compared to Full KV inference while achieving nearly lossless performance.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 26 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

17 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  2. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  4. 2024On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference9 Feb
  5. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  8. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  9. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  10. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  11. 2024Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspectivecited by 13
  12. 2024MoDification: Mixture of Depths Made Easycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  2. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
  3. 2025SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matchingcited by 6
  4. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 5
  5. 2026State Rank Dynamics in Linear Attention LLMscited by 3
  6. 2026CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLMcited by 2
  7. 2026CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  2. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  3. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  4. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  5. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing
  6. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  7. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  8. 2025KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache