mlsanityData to 14 Oct 2026

KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache

arXiv 2509.00579 ↗First posted 30 Aug 2025cs.DC cs.AIcited by 7

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Transformer-based large language models (LLMs) demonstrate impressive potential in various practical applications. However, long context inference poses a significant challenge due to the enormous memory requirements of the key-value (KV) cache, which can scale to multiple gigabytes as sequence length and batch size increase. In this paper, we present KVComp, a generic and efficient KV cache management framework optimized for long-text generation that synergistically works with both latency-critical and throughput-critical inference systems. KVComp employs novel lossy compression techniques specifically designed for KV cache data characteristics, featuring careful co-design of compression algorithms and system architecture. Our approach maintains compatibility with the growing nature of KV cache while preserving high computational efficiency. Experimental results show that KVComp achieves on average 47% and up to 83% higher memory reduction rate compared to existing methods with little/no model accuracy degradation. Furthermore, KVComp achieves extremely high execution throughput, effectively reducing decompression overhead and, in some cases, even accelerating the matrix-vector multiplication operation and outperform cuBLAS-based attention kernels with less data movement.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 39 references that are in the map. First posted on arXiv 30 Aug 2025.

Builds on

18 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2022Galactica: A Large Language Model for Sciencecited by 744
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  11. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  12. 2020cuSZ: An Efficient GPU-Based Error-Bounded Lossy Compression Framework for Scientific Datacited by 18
4 more
  1. 2019DeepSZ: A Novel Framework to Compress Deep Neural Networks by Using Error-Bounded Lossy Compressioncited by 11
  2. 2021COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compressioncited by 8
  3. 2015Dissecting GPU Memory Hierarchy through Microbenchmarkingcited by 6
  4. 2023FZ-GPU: A Fast and High-Ratio Lossy Compressor for Scientific Computing Applications on GPUscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 10
  2. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Servingcited by 7
  3. 2026FibQuant: Universal Vector Quantization for Random-Access KV-Cache Compressioncited by 2
  4. 2026PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantizationcited by 2
  5. 2026Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloadscited by 0
  6. 2026D-Quant: Driftable Entropy Coding for KV Cache Quantizationcited by 0
  7. 2026AlignPred: Error-Controlled KV Cache Compression via Alignment and Predictive Codingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  2. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  3. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  4. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
  5. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  6. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  7. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  8. 2024ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty