mlsanityData to 14 Oct 2026

TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate

arXiv 2504.19874 ↗First posted 28 Apr 2025cs.LG cs.AI cs.DBcited by 125

Not on the map. Why?

Cited by 64 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vector quantization, a problem rooted in Shannon's source coding theory, aims to quantize high-dimensional Euclidean vectors while minimizing distortion in their geometric structure. We propose TurboQuant to address both mean-squared error (MSE) and inner product distortion, overcoming limitations of existing methods that fail to achieve optimal distortion rates. Our data-oblivious algorithms, suitable for online applications, achieve near-optimal distortion rates (within a small constant factor) across all bit-widths and dimensions. TurboQuant achieves this by randomly rotating input vectors, inducing a concentrated Beta distribution on coordinates, and leveraging the near-independence property of distinct coordinates in high dimensions to simply apply optimal scalar quantizers per each coordinate. Recognizing that MSE-optimal quantizers introduce bias in inner product estimation, we propose a two-stage approach: applying an MSE quantizer followed by a 1-bit Quantized JL (QJL) transform on the residual, resulting in an unbiased inner product quantizer. We also provide a formal proof of the information-theoretic lower bounds on best achievable distortion rate by any vector quantizer, demonstrating that TurboQuant closely matches these bounds, differing only by a small constant (≈ 2.7) factor. Experimental results validate our theoretical findings, showing that for KV cache quantization, we achieve absolute quality neutrality with 3.5 bits per channel and marginal quality degradation with 2.5 bits per channel. Furthermore, in nearest neighbor search tasks, our method outperforms existing product quantization techniques in recall while reducing indexing time to virtually zero.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Builds on 48 of its 69 references that are in the map. First posted on arXiv 28 Apr 2025.

Builds on

48 of 69 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
36 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  3. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  4. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  6. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  10. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  11. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  12. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  13. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
  14. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  15. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  16. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  17. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  18. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  19. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  20. 2016A Survey on Learning to Hashcited by 119
  21. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  22. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  23. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  24. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  25. 2023Training Transformers with 4-bit Integerscited by 60
  26. 2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 55
  27. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 52
  28. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 46
  29. 2024Fast Matrix Multiplications for Lookup Table-Quantized LLMscited by 29
  30. 2025PolarQuant: Quantizing KV Caches with Polar Transformationcited by 27
  31. 2024SubGen: Token Generation in Sublinear Time and Memorycited by 21
  32. 2024Practical and Asymptotically Optimal Quantization of High-Dimensional Vectors in Euclidean Space for Approximate Nearest Neighbor Searchcited by 21
  33. 2024Pushing the Limits of Large Language Model Quantization via the Linearity Theoremcited by 21
  34. 2024Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionariescited by 17
  35. 2025Streaming Attention Approximation via Discrepancy Theorycited by 11
  36. 2019Efficient Inner Product Approximation in Hybrid Spacescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 125.

125 papers.

Sort:

By area:

  1. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationcited by 30
  2. 2026ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPUcited by 16
  3. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 9
  4. 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 9
  5. 2026A Note on TurboQuant and the Earlier DRIVE/EDEN Line of Workcited by 8
  6. 2026Provable Quantization with Randomized Hadamard Transformcited by 8
  7. 2026Value-Aware Stochastic KV Cache Eviction for Reasoning Modelscited by 8
  8. 2026High-Rate Quantized Matrix Multiplication Icited by 7
  9. 2026Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Provencited by 6
  10. 2026OrbitQuant: Data-Agnostic Quantization for Image and Video Diffusion Transformerscited by 6
  11. 2026DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantizationcited by 6
  12. 2026SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Servingcited by 5
38 more
  1. 2026DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformerscited by 5
  2. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  3. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  4. 2026Hierarchical vs. Flat Iteration in Shared-Weight Transformerscited by 4
  5. 2026Nearly Optimal Attention Coresetscited by 4
  6. 2026PEEK: Context Map as an Orientation Cache for Long-Context LLM Agentscited by 4
  7. 2026Runtime-Certified Bounded-Error Quantized Attentioncited by 4
  8. 2026The Quantization Benefits of Residual-Free Transformerscited by 4
  9. 2026InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Modelscited by 3
  10. 2026RAD-LAD: Rule and Language Grounded Autonomous Driving in Real-Timecited by 3
  11. 2026Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experimentscited by 3
  12. 2026RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theorycited by 3
  13. 2026TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantizationcited by 2
  14. 2026IsoQuant: Hardware-Aligned SO(4) Isoclinic Rotations for LLM KV Cache Compressioncited by 2
  15. 2026PolarQuant: Optimal Gaussian Weight Quantization via Hadamard Rotation for LLM Compressioncited by 2
  16. 2026SuperLocalMemory V3.3: The Living Brain -- Biologically-Inspired Forgetting, Cognitive Quantization, and Multi-Channel Retrieval for Zero-LLM Agent Memory Systemscited by 2
  17. 2026Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Siliconcited by 2
  18. 2026Approximating Uniform Random Rotations by Two-Stage Structured Hadamard Rotations in High Dimensionscited by 2
  19. 2026eOptShrinkQ: Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantizationcited by 2
  20. 2026Design Conductor 2.0: An agent builds a TurboQuant inference accelerator in 80 hourscited by 2
  21. 2026Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuantcited by 2
  22. 2026FibQuant: Universal Vector Quantization for Random-Access KV-Cache Compressioncited by 2
  23. 2026OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under optimal Squared error quantizationcited by 2
  24. 2026Quantized Keys Steal Attention: Bias Correction for KV-Cache Compression in Video Diffusioncited by 2
  25. 2026Output-Aware Rotation for INT2 KV-Cache Quantizationcited by 2
  26. 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transformscited by 2
  27. 2026Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoningcited by 2
  28. 2026SuperLocalMemory 4.0: The Governed Memory Operating System for AI Agentscited by 2
  29. 2026AXELRAM: Quantize Once, Never Dequantizecited by 1
  30. 2026MoEITS: A Green AI approach for simplifying MoE-LLMscited by 1
  31. 2026Sequential KV Cache Compression via Probabilistic Language Tries: Beyond the Per-Vector Shannon Limitcited by 1
  32. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restorationcited by 1
  33. 2026PrismQuant: Rate-Distortion-Optimal Vector Quantization for Gaussian-Mixture Sourcescited by 1
  34. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reusecited by 1
  35. 2026When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compressioncited by 1
  36. 2026OffQ: Taming Structured Outliers in LLM Quantization by Offsettingcited by 1
  37. 2026APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancingcited by 1
  38. 2026RoPE-Aware Bit Allocation for KV-Cache Quantizationcited by 1

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Fast-TurboQuant: A Multiplier-Free Online Vector Quantization Approach
  2. 2026RQ-MoE: Residual Quantization via Mixture of Experts for Efficient Input-Dependent Vector Compression
  3. 2026Generalized Radius and Integrated Codebook Transforms for Differentiable Vector Quantization
  4. 2023Finite Scalar Quantization: VQ-VAE Made Simple
  5. 2025DiVeQ: Differentiable Vector Quantization Using the Reparameterization Trick
  6. 2026XOR-Trellis: Ultra-Low-Complexity Dequantization and Curvature-Aware Hadamard-Free LLM Quantization
  7. 2026Beyond Stationarity: Rethinking Codebook Collapse in Vector Quantization
  8. 2024Addressing Representation Collapse in Vector Quantized Models with One Linear Layer