KVSculpt: KV Cache Compression as Distillation
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
KV cache compression is critical for efficient long-context LLM inference. Approaches that reduce the per-pair footprint -- quantization and low-rank decomposition -- are orthogonal to those that reduce the sequence length of the cache. Along the sequence-length dimension, existing methods range from pure eviction -- selecting which KV pairs to keep -- to merging, which combines similar pairs into fewer ones. Both remain anchored to the original cache entries. We propose KVSculpt, which moves to the other end of this spectrum: instead of selecting or combining original pairs, we optimize a smaller set of unconstrained KV pairs in continuous embedding space to preserve each layer's attention behavior. Keys are optimized via L-BFGS and values are solved in closed form via least squares, alternating every few steps. On top of this, we introduce adaptive budget allocation, which uses a cheap pilot compression run to redistribute the compression budget across layers and KV heads based on per-component difficulty. On Qwen2.5-1.5B-Instruct with 2048-token contexts, KVSculpt reduces KL divergence by 3.5-4.1x compared to Select+Fit -- attention-score eviction with least-squares value fitting -- across compression ratios r in {0.3, 0.5, 0.7}. Adaptive allocation provides an additional 1.3x KL reduction at no extra inference cost. Analysis reveals that compression difficulty is highly non-uniform: per-layer pilot MSE varies by up to 100x across layers, and the two KV heads within a single layer can differ by up to 467x -- demonstrating that fine-grained budget allocation is essential.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 14 of its 23 references that are in the map. First posted on arXiv 29 Mar 2026.
Builds on
14 of 23 references.
Closest ancestors · 5
Same direction, newest first
- 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
- 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
- 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
- 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May
Other references · 9
- 2017Attention Is All You Needcited by 60,361
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
- 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
- 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
- 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
- 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
- 2026AnchorKV: Anchor-Residual KV Cache Compression
- 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
- 2026MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
- 2025ReFreeKV: Towards Threshold-Free KV Cache Compression
- 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference