mlsanityData to 14 Oct 2026

SQuat: Subspace-orthogonal KV Cache Quantization

arXiv 2503.24358 ↗First posted 31 Mar 2025cs.LG cs.AI cs.CLcited by 7

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

The key-value (KV) cache accelerates LLMs decoding by storing KV tensors from previously generated tokens. It reduces redundant computation at the cost of increased memory usage. To mitigate this overhead, existing approaches compress KV tensors into lower-bit representations; however, quantization errors can accumulate as more tokens are generated, potentially resulting in undesired outputs. In this paper, we introduce SQuat (Subspace-orthogonal KV cache quantization). It first constructs a subspace spanned by query tensors to capture the most critical task-related information. During key tensor quantization, it enforces that the difference between the (de)quantized and original keys remains orthogonal to this subspace, minimizing the impact of quantization errors on the attention mechanism's outputs. SQuat requires no model fine-tuning, no additional calibration dataset for offline learning, and is grounded in a theoretical framework we develop. Through numerical experiments, we show that our method reduces peak memory by 2.17 to 2.82, improves throughput by 2.45 to 3.60, and achieves more favorable benchmark scores than existing KV cache quantization algorithms.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 61 of its 82 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

61 of 82 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations17 Oct
  2. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead5 Jun
  3. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification23 May
  4. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models10 May
  5. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan
  8. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 53

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
38 more
  1. 2020Rethinking Attention with Performerscited by 1,601
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  10. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  13. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  14. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  15. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  16. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  17. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  18. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  19. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  20. 2015Deep Learning with Limited Numerical Precisioncited by 376
  21. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  22. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  23. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  24. 2021Learned Token Pruning for Transformerscited by 139
  25. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  26. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  27. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 96
  28. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  29. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  30. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 68
  31. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 59
  32. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 57
  33. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
  34. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 49
  35. 2024Effectively Compress KV Heads for LLMcited by 30
  36. 2024Pushing the Limits of Large Language Model Quantization via the Linearity Theoremcited by 21
  37. 2024Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionariescited by 17
  38. 2022Beyond Adult and COMPAS: Fairness in Multi-Class Predictioncited by 16

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelitycited by 10
  2. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  3. 2026InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Modelscited by 3
  4. 2026RoPE-Aware Bit Allocation for KV-Cache Quantizationcited by 1
  5. 2026KV Cache Compression Through the Lens of Transform Codingcited by 1
  6. 2025Hopscotch: Discovering and Skipping Redundancies in Language Modelscited by 0
  7. 2026When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  2. 2024Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression
  3. 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
  4. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
  5. 2025CommVQ: Commutative Vector Quantization for KV Cache Compression
  6. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction
  7. 2026E8 Lattice Key Quantization for Low-Bit KV-Cache Compression
  8. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs