mlsanityData to 7 Oct 2026

PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention

arXiv 2609.33746 ↗First posted 27 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

At each decoding step a language model attends over the key-value (KV) cache of every earlier token, so at long context the attention call is bounded by memory bandwidth. Sparse attention reads only a subset of keys chosen by a cheap score estimate, and most methods give the unread tokens zero weight. The output then draws on only a small fraction of the KV cache, and accuracy drops at small budgets, most on tasks that aggregate information across the context. An inverted-file product-quantization (IVF-PQ) index over the cached keys computes an approximate score for every indexed token in order to rank them; after ranking, those scores approximate the attention logits of the tokens left out. PQ-HSA (hybrid sparse-approximate attention) attends the selected tokens with their original keys and values, and the unselected tokens, the background, enter the same softmax through those scores, summed per inverted list and multiplied by the list's mean value. At 128K and a 1-2% retrieval budget, PQ-HSA is more accurate than Quest and SnapKV on Llama-3.1-8B and Qwen3-30B-A3B and stays close to full attention in macro accuracy; with the same selector, the background term raises macro accuracy on the 8B model from 0.71 to 0.83. In the same 128K setting, inside vLLM on one NVIDIA H20, the decode attention call runs 1.6x faster than the FlashAttention-3 kernel; the speedup grows with context length, and a cost model fitted on 8B to 30B models gives the context length at which it begins. A vLLM plugin runs PQ-HSA on two engine versions without changes to the engine source; code is available at https://github.com/KunmingSHAO/pqhsa_release.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

39 of 46 references.

References · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  12. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
27 more
  1. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  2. 2024The Faiss librarycited by 370
  3. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  5. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  6. 2025Qwen2.5-1M Technical Reportcited by 158
  7. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  8. 2020Fast Transformers with Clustered Attentioncited by 123
  9. 2024Transformers are Multi-State RNNscited by 120
  10. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  11. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  12. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  13. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  14. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  15. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compressioncited by 65
  16. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  17. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  18. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  19. 2025CommVQ: Commutative Vector Quantization for KV Cache Compressioncited by 25
  20. 2024HashAttention: Semantic Sparsity for Faster Inferencecited by 23
  21. 2025Multipole Attention for Efficient Long Context Reasoningcited by 14
  22. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  23. 2019Norm-Explicit Quantization: Improving Vector Quantization for Maximum Inner Product Searchcited by 10
  24. 2026Flash-KMeans: Fast and Memory-Efficient Exact K-Meanscited by 7
  25. 2026Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keyscited by 5
  26. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMscited by 2
  27. 2026STS: Efficient Sparse Attention with Speculative Token Sparsitycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Rectified Sparse Attention
  2. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity
  3. 2026Simplified Sparse Attention via Gist Tokens
  4. 2026RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
  5. 2026Scaling Attention via Feature Sparsity
  6. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
  7. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
  8. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving