mlsanityData to 14 Oct 2026

KV Cache Compression Through the Lens of Transform Coding

arXiv 2608.14191 ↗First posted 14 Aug 2026cs.LG cs.CL eess.SPcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference. Existing quantization methods address this bottleneck by representing the KV cache uniformly with lower-precision data types and designing quantization schemes to minimize reconstruction error in the cache itself, without accounting for how that error propagates through attention mechanisms. We prove that, under a white-noise quantization model, the expected attention-aware distortion decomposes into additive key and value contributions that factor across tokens and channels. Building on transform coding and reverse water-filling, which are classical tools from signal processing and rate-distortion theory, we introduce Attention-Aware Transform Coding (AATC), which allocates bits over a calibration set to minimize attention-aware distortion. On Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct, evaluated across LongBench, RULER, GSM8K, MMLU-Pro, and MATH-500, our method achieves near-lossless accuracy at approximately 5.8× compression, whereas each baseline degrades in at least some settings.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 51 references that are in the map. First posted on arXiv 14 Aug 2026.

Builds on

38 of 51 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026eOptShrinkQ: Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantization6 Apr
  2. 2025KV Cache Transform Coding for Compact Storage in LLM Inference3 Nov
  3. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  4. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction24 Mar
  5. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective6 Feb
  6. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  7. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
  8. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters18 Jun
  9. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
15 more
  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  4. 2021A White Paper on Neural Network Quantizationcited by 472
  5. 2020Nonlinear Transform Codingcited by 136
  6. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  7. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  8. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 46
  9. 2025CommVQ: Commutative Vector Quantization for KV Cache Compressioncited by 28
  10. 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 9
  11. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoningcited by 8
  12. 2025A2ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantizationcited by 7
  13. 2025SQuat: Subspace-orthogonal KV Cache Quantizationcited by 7
  14. 2026Runtime-Certified Bounded-Error Quantized Attentioncited by 4
  15. 2026RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theorycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Low-Bit Recurrent States in Hybrid Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
  2. 2026The risk of KV cache compression
  3. 2026WUSH-KV: KV Cache Quantization with Data-Adaptive Transforms
  4. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
  5. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
  6. 2026Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration
  7. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  8. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs