mlsanityData to 7 Oct 2026

QUOKA: Query-Oriented KV Selection For Efficient LLM Prefill

arXiv 2602.08722 ↗First posted 9 Feb 2026cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

We present QUOKA: Query-oriented KV selection for efficient attention, a training-free and hardware agnostic sparse attention algorithm for accelerating transformer inference under chunked prefill. While many queries focus on a smaller group of keys in the attention operator, we observe that queries with low cosine similarity with respect to the mean query interact more strongly with more keys and have the greatest contribution to final attention logits. By prioritizing these low cosine similarity queries, the behavior of full attention during the prefill stage can be closely approximated. QUOKA leverages this observation, accelerating attention by (1) first retaining a small set of representative queries and (2) then subselectin the keys most aligned with those queries. Through experiments on Needle-In-A-Haystack, LongBench, RULER, and Math500, we show that, while realizing a 3x reduction in time-to-first-token, 5x speedup in attention on Nvidia GPUs and up to nearly a 7x speedup on Intel Xeon CPUs, QUOKA achieves near-baseline accuracy, utilizing 88% fewer key-value pairs per attention evaluation.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 36 of its 41 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

36 of 41 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
24 more
  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  5. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  7. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  8. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  9. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  10. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  11. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  12. 2024Transformers are Multi-State RNNscited by 120
  13. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  14. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  15. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  16. 2019Explicit Sparse Transformer: Concentrated Attention Through Explicit Selectioncited by 61
  17. 2024Fast On-device LLM Inference with NPUscited by 58
  18. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  19. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  20. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  21. 2021Leveraging redundancy in attention with Reuse Transformerscited by 33
  22. 2024TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attentioncited by 28
  23. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  24. 2024Cottention: Linear Transformers With Cosine Attentioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compressioncited by 2
  2. 2026CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selectioncited by 0
  3. 2026RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reusecited by 0