mlsanityData to 7 Oct 2026

GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models

arXiv 2601.04719 ↗First posted 8 Jan 2026cs.LG cs.PFcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

The key-value (KV) cache in large language models presents a significant memory bottleneck during inference, growing linearly with sequence length and often exceeding the memory footprint of model weights themselves. We implement and evaluate GPU-accelerated INT8 quantization for KV cache compression, achieving 4× memory reduction with minimal accuracy degradation. We develop four CUDA kernel variants -- naive, tiled, coarsened, and vectorized -- and benchmark them across realistic workload sizes up to 1 billion elements. Our vectorized kernel achieves up to 1,694× speedup over CPU baselines while maintaining reconstruction error below 0.004 and attention score error below 0.1 even for 8K-dimensional heads. These results demonstrate that INT8 quantization provides a practical approach for reducing memory pressure in LLM inference with negligible computational overhead (6--58ms) and minimal impact on downstream model behavior

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 10 of its 10 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

10 of 10 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 8

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  8. 2022Efficiently Scaling Transformer Inferencecited by 529

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
  2. 2026InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models
  3. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
  4. 2026Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
  5. 2025CommVQ: Commutative Vector Quantization for KV Cache Compression
  6. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
  7. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference
  8. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference