mlsanityData to 7 Oct 2026

Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation

arXiv 2609.04263 ↗First posted 2 Sep 2026cs.AR cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Low-bit key--value (KV) caches reduce the memory required for autoregressive decoding, but the resulting quality loss depends on the model and quantizer. We keep the quantizer fixed and distill the floating-cache model's behavior into low-rank Q/K/V projection updates while the student executes a physically packed incremental cache. Across three seeds, 4-bit affine-cache adapters recover 54.24%±2.47% of the held-out perplexity gap on TinyLlama-1.1B and 75.96%±4.04% on Gemma-4-12B. On the same frozen NF4 Llama-3.1-8B base, one validation-selected run per quantizer recovers 60.42% under KIVI K2V2 and 37.61% under KVarN K4V2, while preserving 180-case associative retrieval. Gemma's score on an official 4K/8K RULER subset rises from 42.80 with the unadapted 4-bit cache to 48.33 after adaptation (46.15 floating), with substantial task heterogeneity. Finally, a 2-bit rank--token sweep reduces TinyLlama's 2-bit PPL from 576.10 to 11.4000±0.0059 across three seeds, versus 10.3988 floating, but restores only 11--12 of 180 retrieval cases. These results show that low-rank projection adaptation can recover held-out quality across fixed cache formats, while perplexity recovery need not restore long-context retrieval.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 16 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

15 of 16 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks2 Jun
  2. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction6 Oct
  3. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference6 Feb
  4. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations25 Jan
  5. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models10 May
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 8

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  6. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  7. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  8. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Controlcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
  2. 2026Read What Matters: Query-Adaptive Quantization for KV Caches
  3. 2026Runtime-Certified Bounded-Error Quantized Attention
  4. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  5. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
  6. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
  7. 2026KV Cache Compression Through the Lens of Transform Coding
  8. 2026Attention-Weighted Value Projection for KV-Cache Compression