mlsanityData to 14 Oct 2026

KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization

arXiv 2405.03917 ↗First posted 7 May 2024cs.LGcited by 76

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Efficient deployment of Large Language Models (LLMs) requires batching multiple requests together to improve throughput. As the batch size, context length, or model size increases, the size of the key and value (KV) cache can quickly become the main contributor to GPU memory usage and the bottleneck of inference latency. Quantization has emerged as an effective technique for KV cache compression, but existing methods still fail at very low bit widths. We observe that distinct channels of a key/value activation embedding are highly inter-dependent, and the joint entropy of multiple channels grows at a slower rate than the sum of their marginal entropies. Based on this insight, we propose Coupled Quantization (CQ), which couples multiple key/value channels together to exploit their inter-dependency and encode the activations in a more information-efficient manner. Extensive experiments reveal that CQ outperforms or is competitive with existing baselines in preserving model quality. Furthermore, we demonstrate that CQ can preserve model quality with KV cache quantized down to 1-bit.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 51 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

34 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
20 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2023Mistral 7Bcited by 3,164
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  12. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  13. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  14. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  15. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  16. 2023Challenges and Applications of Large Language Modelscited by 254
  17. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  18. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  19. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  20. 2024NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attentioncited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 30; 2026: 28.

76 papers.

Sort:

By area:

  1. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  2. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 131
  3. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  4. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 96
  5. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 52
  6. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 51
  7. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 42
  8. 2024Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compressioncited by 33
  9. 2024Fast State Restoration in LLM Serving with HCachecited by 33
  10. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 31
  11. 2025CommVQ: Commutative Vector Quantization for KV Cache Compressioncited by 28
  12. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 24
38 more
  1. 2024LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Gridcited by 22
  2. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 17
  3. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  4. 2024UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspectivecited by 15
  5. 2025Don't Pass@k: A Bayesian Framework for Large Language Model Evaluationcited by 15
  6. 2025Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Modelscited by 14
  7. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Headcited by 13
  8. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Modelscited by 13
  9. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  10. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Expertscited by 12
  11. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  12. 2025Streaming Attention Approximation via Discrepancy Theorycited by 11
  13. 2025SALS: Sparse Attention in Latent Space for KV cache Compressioncited by 11
  14. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compressioncited by 10
  15. 2025Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boostcited by 10
  16. 2026Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategiescited by 9
  17. 2025BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cachecited by 8
  18. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 8
  19. 2024FutureFill: Fast Generation from Convolutional Sequence Modelscited by 7
  20. 2024Lossless KV Cache Compression to 2%cited by 7
  21. 2024Residual vector quantization for KV cache compression in large language modelcited by 7
  22. 2025NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cachecited by 7
  23. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  24. 2024QCQA: Quality and Capacity-aware grouped Query Attentioncited by 6
  25. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiencycited by 6
  26. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Designcited by 5
  27. 2025FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantizationcited by 5
  28. 2025To Compress or Not? Pushing the Frontier of Lossless GenAI Model Weights Compression with Exponent Concentrationcited by 5
  29. 2026Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keyscited by 5
  30. 2026SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Servingcited by 5
  31. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantizationcited by 4
  32. 2025Efficient Low Rank Attention for Long-Context Inference in Large Language Modelscited by 4
  33. 2026The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inferencecited by 4
  34. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Servingcited by 4
  35. 2026What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agentscited by 4
  36. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  37. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inferencecited by 4
  38. 2024Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptationcited by 3

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models
  2. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  3. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  4. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
  5. 2025H1B-KV: Hybrid One-Bit Caches for Memory-Efficient Large Language Model Inference
  6. 2026SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Coding
  7. 2026GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models
  8. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization