mlsanityData to 7 Oct 2026

PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression

arXiv 2512.24449 ↗First posted 30 Dec 2025cs.DC cs.AIcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Transformer-based large language models (LLMs) have demonstrated remarkable potential across a wide range of practical applications. However, long-context inference remains a significant challenge due to the substantial memory requirements of the key-value (KV) cache, which can scale to several gigabytes as sequence length and batch size increase. In this paper, we present PackKV, a generic and efficient KV cache management framework optimized for long-context generation. %, which synergistically supports both latency-critical and throughput-critical inference scenarios. PackKV introduces novel lossy compression techniques specifically tailored to the characteristics of KV cache data, featuring a careful co-design of compression algorithms and system architecture. Our approach is compatible with the dynamically growing nature of the KV cache while preserving high computational efficiency. Experimental results show that, under the same and minimum accuracy drop as state-of-the-art quantization methods, PackKV achieves, on average, 153.2% higher memory reduction rate for the K cache and 179.6% for the V cache. Furthermore, PackKV delivers extremely high execution throughput, effectively eliminating decompression overhead and accelerating the matrix-vector multiplication operation. Specifically, PackKV achieves an average throughput improvement of 75.7% for K and 171.7% for V across A100 and RTX Pro 6000 GPUs, compared to cuBLAS matrix-vector multiplication kernels, while demanding less GPU memory bandwidth. Code available on https://github.com/BoJiang03/PackKV

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 37 references that are in the map. First posted on arXiv 30 Dec 2025.

Builds on

20 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  6. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  7. 2022Galactica: A Large Language Model for Sciencecited by 736
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  12. 2022Efficiently Scaling Transformer Inferencecited by 529
5 more
  1. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  2. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  3. 2019DeepSZ: A Novel Framework to Compress Deep Neural Networks by Using Error-Bounded Lossy Compressioncited by 11
  4. 2021COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compressioncited by 8
  5. 2015Dissecting GPU Memory Hierarchy through Microbenchmarkingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inferencecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache
  2. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  3. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
  4. 2024ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty
  5. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  6. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
  7. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  8. 2026EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction