mlsanityData to 7 Oct 2026

WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More

arXiv 2402.12065 ↗First posted 19 Feb 2024cs.LG cs.AI cs.CLcited by 69

Not on the map. Why?

Cited by 42 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) face significant deployment challenges due to their substantial memory requirements and the computational demands of auto-regressive text generation process. This paper addresses these challenges by focusing on the quantization of LLMs, a technique that reduces memory consumption by converting model parameters and activations into low-bit integers. We critically analyze the existing quantization approaches, identifying their limitations in balancing the accuracy and efficiency of the quantized LLMs. To advance beyond these limitations, we propose WKVQuant, a PTQ framework especially designed for quantizing weights and the key/value (KV) cache of LLMs. Specifically, we incorporates past-only quantization to improve the computation of attention. Additionally, we introduce two-dimensional quantization strategy to handle the distribution of KV cache, along with a cross-block reconstruction regularization for parameter optimization. Experiments show that WKVQuant achieves almost comparable memory savings to weight-activation quantization, while also approaching the performance of weight-only quantization.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

24 of 35 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
12 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  3. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  5. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  6. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  7. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  8. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  9. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 173
  10. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  11. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  12. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 30; 2026: 15.

69 papers.

Sort:

By area:

  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  4. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  5. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  6. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  7. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 59
  8. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 59
  9. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 52
  10. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 47
  11. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  12. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 41
38 more
  1. 2025MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantizationcited by 35
  2. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
  3. 2024A Survey of Small Language Modelscited by 31
  4. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30
  5. 2025MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methodscited by 26
  6. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 26
  7. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  8. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 21
  9. 2025GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuningcited by 21
  10. 2024I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Modelscited by 19
  11. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurationscited by 19
  12. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMscited by 19
  13. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 18
  14. 2024Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionariescited by 17
  15. 2025ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibrationcited by 16
  16. 2025Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modelingcited by 15
  17. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Modelscited by 12
  18. 2024Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Mannercited by 10
  19. 2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compressioncited by 10
  20. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
  21. 2025Streaming Attention Approximation via Discrepancy Theorycited by 9
  22. 2025Quantize What Counts: More for Keys, Less for Valuescited by 9
  23. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correctioncited by 9
  24. 2024BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inferencecited by 8
  25. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  26. 2025SQuat: Subspace-orthogonal KV Cache Quantizationcited by 7
  27. 2026AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completioncited by 7
  28. 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 7
  29. 2024COMET: Towards Partical W4A4KV4 LLMs Servingcited by 6
  30. 2024Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQAcited by 6
  31. 2025PCDVQ: Enhancing Vector Quantization for Large Language Models via Polar Coordinate Decouplingcited by 5
  32. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 5
  33. 2025Interpreting the Effects of Quantization on LLMscited by 5
  34. 2025Lethe: Layer- and Time-Adaptive KV Cache Pruning for Reasoning-Intensive LLM Servingcited by 5
  35. 2026P-EAGLE: Parallel-Drafting EAGLE with Scalable Trainingcited by 5
  36. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMscited by 5
  37. 2024AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantizationcited by 4
  38. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cachecited by 4

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  2. 2025XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
  4. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
  5. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
  6. 2024QAQ: Quality Adaptive Quantization for LLM KV Cache
  7. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  8. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs