mlsanityData to 7 Oct 2026

PQCache: Product Quantization-based KVCache for Long Context LLM Inference

arXiv 2407.12820 ↗First posted 1 Jul 2024cs.CL cs.AI cs.LGcited by 93

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), the intermediate representations of tokens within LLM inference, has now become the primary memory bottleneck due to limited GPU memory. Current methods selectively determine suitable keys and values for self-attention computation in LLMs to address the issue. However, they either fall short in maintaining model quality or result in high serving latency. Drawing inspiration from advanced embedding retrieval techniques prevalent in the data management community, we consider the storage and retrieval of KVCache as a typical embedding retrieval problem. We propose PQCache, which employs Product Quantization (PQ) to manage KVCache, maintaining model quality while ensuring low serving latency. During the prefilling phase, we apply PQ to tokens' keys for each LLM layer and head. During the autoregressive decoding phase, we use PQ codes and centroids to approximately identify important preceding tokens, then fetch the corresponding key-value pairs for self-attention computation. Through meticulous design of overlapping and caching, we minimize any additional computation and communication overhead during both phases. Extensive experiments demonstrate that PQCache achieves both effectiveness and efficiency, with 4.60% score improvement over existing methods on InfiniteBench and low system latency in both prefilling and decoding.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 57 of its 121 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

57 of 121 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024QAQ: Quality Adaptive Quantization for LLM KV Cache7 Mar
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 54

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
38 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  2. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  7. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  10. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  11. 2023A Survey on Large Language Models for Recommendationcited by 378
  12. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  13. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  14. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  15. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  16. 2023Large Language Models in Finance: A Surveycited by 223
  17. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  18. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  19. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  20. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  21. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  22. 2019Convergence of Edge Computing and Deep Learning: A Comprehensive Surveycited by 120
  23. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  24. 2023The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4cited by 112
  25. 2017Fast Approximate Nearest Neighbor Search With The Navigating Spreading-out Graphcited by 103
  26. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  27. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  28. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 99
  29. 2021A Comprehensive Survey and Experimental Comparison of Graph-Based Approximate Nearest Neighbor Searchcited by 82
  30. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  31. 2019Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systemscited by 68
  32. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  33. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  34. 2018Revisiting the Inverted Indices for Billion-Scale Approximate Nearest Neighborscited by 37
  35. 2024On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inferencecited by 32
  36. 2021Survey: Transformer based Video-Language Pre-trainingcited by 27
  37. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  38. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budgetcited by 21

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 34; 2026: 47.

93 papers.

Sort:

By area:

  1. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  4. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  5. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  6. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  7. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  8. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  9. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  10. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  11. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  12. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
38 more
  1. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  2. 2026Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantizationcited by 18
  3. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  4. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  5. 2025Don't Pass@k: A Bayesian Framework for Large Language Model Evaluationcited by 15
  6. 2025Multipole Attention for Efficient Long Context Reasoningcited by 14
  7. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  8. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruningcited by 13
  9. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  10. 2025Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decodingcited by 9
  11. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  12. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  13. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 8
  14. 2025PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Accelerationcited by 7
  15. 2025AttentionPredictor: Temporal Patterns Matter for KV Cache Compressioncited by 7
  16. 2025A2ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantizationcited by 7
  17. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  18. 2024KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Modelscited by 6
  19. 2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attentioncited by 6
  20. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Servingcited by 6
  21. 2025SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matchingcited by 6
  22. 2025vAttention: Verified Sparse Attentioncited by 6
  23. 2025Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUscited by 5
  24. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 5
  25. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMscited by 5
  26. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attentioncited by 5
  27. 2025X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compressioncited by 4
  28. 2025LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequencescited by 4
  29. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limitscited by 4
  30. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMscited by 4
  31. 2026Runtime-Certified Bounded-Error Quantized Attentioncited by 4
  32. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefillingcited by 3
  33. 2025AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Modelscited by 3
  34. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
  35. 2026SOCKET: SOft Collision Kernel EsTimator for Sparse Attentioncited by 3
  36. 2026DashAttention: Differentiable and Adaptive Sparse Hierarchical Attentioncited by 3
  37. 2026RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inferencecited by 3
  38. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inferencecited by 2

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization
  2. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
  3. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
  4. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  5. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs
  6. 2024Effectively Compress KV Heads for LLM
  7. 2026SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference
  8. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference