mlsanityData to 7 Oct 2026

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models

arXiv 2501.19392 ↗First posted 31 Jan 2025cs.LGcited by 21

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Efficient real-world deployments of large language models (LLMs) rely on Key-Value (KV) caching for processing and generating long outputs, reducing the need for repetitive computation. For large contexts, Key-Value caches can take up tens of gigabytes of device memory, as they store vector representations for each token and layer. Recent work has shown that the cached vectors can be compressed through quantization, pruning or merging, but these techniques often compromise quality towards higher compression rates. In this work, we aim to improve Key & Value compression by exploiting two observations: 1) the inherent dependencies between keys and values across different layers, and 2) high-compression mechanisms for internal network states. We propose AQUA-KV, an adaptive quantization for Key-Value caches that relies on compact adapters to exploit existing dependencies between Keys and Values, and aims to "optimally" compress the information that cannot be predicted. AQUA-KV significantly improves compression rates, while maintaining high accuracy on state-of-the-art LLM families. On Llama 3.2 LLMs, we achieve near-lossless inference at 2-2.5 bits per value with under 1% relative error in perplexity and LongBench scores. AQUA-KV is one-shot, simple, and efficient: it can be calibrated on a single GPU within 1-6 hours, even for 70B models.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 54 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

43 of 54 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  2. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  3. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing24 Oct
  4. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  6. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  7. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  11. 2021Ethical and social risks of harm from Language Modelscited by 957
  12. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
21 more
  1. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  3. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  4. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  5. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  6. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  7. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  8. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  9. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 86
  10. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 78
  11. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  12. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  13. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  14. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 59
  15. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
  16. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 47
  17. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 41
  18. 2024Fast Matrix Multiplications for Lookup Table-Quantized LLMscited by 27
  19. 2024Your Transformer is Secretly Linearcited by 23
  20. 2024Pushing the Limits of Large Language Model Quantization via the Linearity Theoremcited by 20
  21. 2024PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 20.

21 papers.

Sort:

By area:

  1. 2026Evaluating Memory Structure in LLM Agentscited by 12
  2. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  3. 2025SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruningcited by 6
  4. 2026Gated Differentiable Working Memory for Long-Context Language Modelingcited by 6
  5. 2026Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategiescited by 6
  6. 2026LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Evictioncited by 4
  7. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cachecited by 4
  8. 2026Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Samplingcited by 4
  9. 2026A Simple Plug-in for Improving Eviction-Based KV Cache Compressioncited by 3
  10. 2026Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compressioncited by 2
  11. 2026Linear Predictability of Attention Heads in Large Language Modelscited by 1
  12. 2026KV Cache Offloading for Context-Intensive Taskscited by 1
9 more
  1. 2026QKVShare: Quantized KV-Cache Handoff for Multi-Agent On-Device LLMscited by 1
  2. 2026RoPE-Aware Bit Allocation for KV-Cache Quantizationcited by 1
  3. 2026Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloadscited by 0
  4. 2026Learning how to Forget: Fine-tuning for Long-Context Sparse Attentioncited by 0
  5. 2026A Target-Centric Survey of Quantization-Aware Trainingcited by 0
  6. 2026KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocationcited by 0
  7. 2026ARM: Attention with Routed-Memory for Learnable Sparse Controlcited by 0
  8. 2026WUSH-KV: KV Cache Quantization with Data-Adaptive Transformscited by 0
  9. 2026AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVDcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
  2. 2026VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
  3. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  4. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  5. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  6. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
  7. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  8. 2025H1B-KV: Hybrid One-Bit Caches for Memory-Efficient Large Language Model Inference