mlsanityData to 7 Oct 2026

QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead

arXiv 2406.03482 ↗First posted 5 Jun 2024cs.LG cs.AI cs.CLcited by 47

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Serving LLMs requires substantial memory due to the storage requirements of Key-Value (KV) embeddings in the KV cache, which grows with sequence length. An effective approach to compress KV cache is quantization. However, traditional quantization methods face significant memory overhead due to the need to store quantization constants (at least a zero point and a scale) in full precision per data block. Depending on the block size, this overhead can add 1 or 2 bits per quantized number. We introduce QJL, a new quantization approach that consists of a Johnson-Lindenstrauss (JL) transform followed by sign-bit quantization. In contrast to existing methods, QJL eliminates memory overheads by removing the need for storing quantization constants. We propose an asymmetric estimator for the inner product of two vectors and demonstrate that applying QJL to one vector and a standard JL transform without quantization to the other provides an unbiased estimator with minimal distortion. We have developed an efficient implementation of the QJL sketch and its corresponding inner product estimator, incorporating a lightweight CUDA kernel for optimized computation. When applied across various LLMs and NLP tasks to quantize the KV cache to only 3 bits, QJL demonstrates a more than fivefold reduction in KV cache memory usage without compromising accuracy, all while achieving faster runtime. Codes are available at urlhttps://github.com/amirzandieh/QJL.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 32 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

23 of 32 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization7 May
  2. 2024QAQ: Quality Adaptive Quantization for LLM KV Cache7 Mar
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
7 more
  1. 2022Efficiently Scaling Transformer Inferencecited by 529
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  3. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  4. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 99
  5. 2016Orthogonal Random Featurescited by 89
  6. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  7. 2024SubGen: Token Generation in Sublinear Time and Memorycited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 12; 2026: 30.

47 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  3. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  4. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 21
  5. 2024A Tighter Complexity Analysis of SparseGPTcited by 19
  6. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  7. 2025Streaming Attention Approximation via Discrepancy Theorycited by 9
  8. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correctioncited by 9
  9. 2025PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Accelerationcited by 7
  10. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  11. 2025SQuat: Subspace-orthogonal KV Cache Quantizationcited by 7
  12. 2026A Note on TurboQuant and the Earlier DRIVE/EDEN Line of Workcited by 7
35 more
  1. 2024HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashingcited by 6
  2. 2026High-Rate Quantized Matrix Multiplication Icited by 6
  3. 2025Compression Barriers for Autoregressive Transformerscited by 5
  4. 2026Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keyscited by 5
  5. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  6. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cachecited by 4
  7. 2025NOSA: Native and Offloadable Sparse Attentioncited by 4
  8. 2026Hierarchical vs. Flat Iteration in Shared-Weight Transformerscited by 4
  9. 2026Runtime-Certified Bounded-Error Quantized Attentioncited by 4
  10. 2025CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMscited by 3
  11. 2026Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experimentscited by 3
  12. 2025Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decodingcited by 2
  13. 2026TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantizationcited by 2
  14. 2026IsoQuant: Hardware-Aligned SO(4) Isoclinic Rotations for LLM KV Cache Compressioncited by 2
  15. 2026Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Siliconcited by 2
  16. 2026Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuantcited by 2
  17. 2026FibQuant: Universal Vector Quantization for Random-Access KV-Cache Compressioncited by 2
  18. 2026OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under optimal Squared error quantizationcited by 2
  19. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  20. 2026SuperLocalMemory 4.0: The Governed Memory Operating System for AI Agentscited by 2
  21. 2026AXELRAM: Quantize Once, Never Dequantizecited by 1
  22. 2026SuperLocalMemory V3.3: The Living Brain -- Biologically-Inspired Forgetting, Cognitive Quantization, and Multi-Channel Retrieval for Zero-LLM Agent Memory Systemscited by 1
  23. 2026HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Modelscited by 1
  24. 20263DTurboQuant: Training-Free Near-Optimal Quantization for 3D Reconstruction Modelscited by 0
  25. 2026IVF-TQ: Calibration-Free Streaming Vector Search via a Codebook-Free Residual Layercited by 0
  26. 2026Block-Sphere Vector Quantizationcited by 0
  27. 2026Hurwitz Quaternion Multiplicative Quantization for KV Cache Compressioncited by 0
  28. 2026EinSort: Sorting is All We Need for Tensorizing LLMcited by 0
  29. 2026UltraQuant: 4-bit KV Caching for Context-Heavy Agentscited by 0
  30. 2026Ablation, Statistical Inference, and Validation for KV-Cache Compressioncited by 0
  31. 2026VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMscited by 0
  32. 2026RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attentioncited by 0
  33. 2026VQ-bench: A Composable Vector Quantization Frameworkcited by 0
  34. 2026RSLM: Training-Free Vector Quantization for Approximate Nearest Neighbor Searchcited by 0
  35. 2026Self-Indexing Attention for Compression-Compatible Sparse Long-Context LLM Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026WUSH-KV: KV Cache Quantization with Data-Adaptive Transforms
  2. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations
  3. 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
  4. 2026D-Quant: Driftable Entropy Coding for KV Cache Quantization
  5. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  6. 2025NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
  7. 2026RoPE-Aware Bit Allocation for KV-Cache Quantization
  8. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Models