mlsanityData to 14 Oct 2026

ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs

arXiv 2503.10714 ↗First posted 13 Mar 2025cs.CL cs.AIcited by 8

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The linear growth of key-value (KV) cache memory and quadratic computational in attention mechanisms complexity pose significant bottlenecks for large language models (LLMs) in long-context processing. While existing KV cache optimization methods address these challenges through token pruning or feature merging, they often incur irreversible information loss or require costly parameter retraining. To this end, we propose ZSMerge, a dynamic KV cache compression framework designed for efficient cache management, featuring three key operations: (1) fine-grained memory allocation guided by multi-dimensional token importance metrics at head-level granularity, (2) a residual merging mechanism that preserves critical context through compensated attention scoring, and (3) a zero-shot adaptation mechanism compatible with diverse LLM architectures without requiring retraining. ZSMerge significantly enhances memory efficiency and inference speed with negligible performance degradation across LLMs. When applied to LLaMA2-7B, it demonstrates a 20:1 compression ratio for key-value cache retention (reducing memory footprint to 5% of baseline) while sustaining comparable generation quality, coupled with triple throughput gains at extreme 54k-token contexts that eliminate out-of-memory failures. The code is available at https://github.com/SusCom-Lab/ZSMerge.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 41 of its 50 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

41 of 50 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  2. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  3. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget7 Apr
  6. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2024Qwen2 Technical Reportcited by 2,120
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
21 more
  1. 2022Token Merging: Your ViT But Fastercited by 1,021
  2. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  3. 2024Yi: Open Foundation Models by 01.AIcited by 719
  4. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  5. 2023The Falcon Series of Open Language Modelscited by 535
  6. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  7. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  8. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  9. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  10. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  11. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  12. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  13. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  14. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  15. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  16. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  17. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  18. 2021On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformerscited by 14
  19. 2024LLaMA-NAS: Efficient Neural Architecture Search for Large Language Modelscited by 9
  20. 2024INT-FlashAttention: Enabling Flash Attention for INT8 Quantizationcited by 9
  21. 2023Self-Explanation Prompting Improves Dialogue Understanding in Large Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2026Cartridges at Scale: Training Modular KV Caches over Large Document Collectionscited by 6
  2. 2026Every Cache Entry Earns Its Place: Global Allocation of Resolution and Coverage for KV Cache Compressioncited by 2
  3. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  4. 2026Joint Encoding of KV-Cache Blocks for Scalable LLM Servingcited by 1
  5. 2026Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memorycited by 1
  6. 2026KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Mergingcited by 0
  7. 2026Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compressioncited by 0
  8. 2026VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
  2. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  3. 2026GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
  4. 2024ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
  6. 2026DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocation
  7. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  8. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs