mlsanityData to 7 Oct 2026

Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost

arXiv 2511.18643 ↗First posted 23 Nov 2025cs.LG cs.AIcited by 8

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

The KV cache is a dominant memory bottleneck for LLM inference. While 4-bit KV quantization preserves accuracy, 2-bit often degrades it, especially on long-context reasoning. We close this gap via an algorithm-system co-design for mixed-precision KV caching: Kitty. On the algorithm side, extensive experiments show that Dynamic Channel-wise Precision Boost -- which ranks Key-cache channels by sensitivity and keeps only a small fraction at higher precision -- maintains near-zero loss in accuracy drop while approaching 2-bit memory. The main challenge is handling dynamic 4-bit channel boosts while keeping the page layout coalesced and the dequantization uniform, with no scattered reads or hard-coded masks. Kitty addresses these issues by decompose each mixed-precision Key page into two tensors with unified 2-bit precision. Based on this, Kitty provides a page-centric KV layout, Triton-compatible page dequantization kernels, and a lightweight runtime pipeline that preserves coalescing and avoids divergence. Across seven tasks and two model families (Qwen3, LLaMA3), Kitty cuts KV memory by nearly 8x with negligible accuracy loss, enabling up to 8x larger batches and 2.1x-4.1x higher throughput under the same memory budget. We release the full implementation of Kitty at https://github.com/Summer-Summer/Kitty.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 29 references that are in the map. First posted on arXiv 23 Nov 2025.

Builds on

20 of 29 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference6 Feb
  2. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization7 May
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 16

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  12. 2020Sparse GPU Kernels for Deep Learningcited by 138
4 more
  1. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 68
  2. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
  3. 2025BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cachecited by 7
  4. 2024MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cachecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 7
  2. 2026KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Taskscited by 6
  3. 2026SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Servingcited by 4
  4. 2026RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theorycited by 2
  5. 2026Output-Aware Rotation for INT2 KV-Cache Quantizationcited by 2
  6. 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transformscited by 2
  7. 2026RoPE-Aware Bit Allocation for KV-Cache Quantizationcited by 1
  8. 2026Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Tailoring the Quantization Space for 1-Bit KV Cache Compression
  2. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
  3. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs
  4. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction
  5. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache
  6. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs
  7. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  8. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization