mlsanityData to 7 Oct 2026

QET: Enhancing Quantized LLM Parameters and KV cache Compression through Element Substitution and Residual Clustering

arXiv 2407.03637 ↗First posted 4 Jul 2024cs.LG cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

The matrix quantization entails representing matrix elements in a more space-efficient form to reduce storage usage, with dequantization restoring the original matrix for use. We formulate the Quantization Error Minimization (QEM) problem as minimizing the distance between a matrix before and after quantization, under the condition that the quantized matrix occupies the same memory space. Matrix quantization is crucial in various applications, including Large Language Models (LLMs) weight quantization, vector databases, KV cache quantization, graph compression, and image compression. Recent advancements in LLMs, such as GPT-4 and BERT, have highlighted the importance of matrix compression due to the large size of parameters and KV cache, which are stored as matrices. We propose Quantum Entanglement Trees (QET) to address the QEM problem by leveraging the local orderliness of matrix elements, involving iterative element swapping to form a locally ordered matrix. This matrix is then grouped and quantized by columns. To enhance QET, we introduce two optimizations: further quantizing residuals to reduce MSE, and using masking and batch processing to accelerate the algorithm. Experimental results demonstrate that QET can effectively reduce MSE to 5.05%, 13.33%, and 11.89% of the current best method on the LLM dataset, K cache, and V cache, respectively. Our contributions include the abstraction of the QEM problem, the design of the QET algorithm, and the proposal of two optimizations to improve accuracy and speed.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 32 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

22 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models10 May
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  11. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  12. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
8 more
  1. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  2. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  4. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  5. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  6. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 20
  7. 2017Online Product Quantizationcited by 11
  8. 2016Scalable Image Retrieval by Sparse Product Quantizationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression
  2. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
  3. 2025CommVQ: Commutative Vector Quantization for KV Cache Compression
  4. 2024Residual vector quantization for KV cache compression in large language model
  5. 2026D-Quant: Driftable Entropy Coding for KV Cache Quantization
  6. 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
  8. 2025Gradual Binary Search and Dimension Expansion : A general method for activation quantization in LLMs