mlsanityData to 7 Oct 2026

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

arXiv 2406.10774 ↗First posted 16 Jun 2024cs.CL cs.LGcited by 487

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

As the demand for long-context large language models (LLMs) increases, models with context windows of up to 128K or 1M tokens are becoming increasingly prevalent. However, long-context LLM inference is challenging since the inference speed decreases significantly as the sequence length grows. This slowdown is primarily caused by loading a large KV cache during self-attention. Previous works have shown that a small portion of critical tokens will dominate the attention outcomes. However, we observe the criticality of a token highly depends on the query. To this end, we propose Quest, a query-aware KV cache selection algorithm. Quest keeps track of the minimal and maximal Key values in KV cache pages and estimates the criticality of a given page using Query vectors. By only loading the Top-K critical KV cache pages for attention, Quest significantly speeds up self-attention without sacrificing accuracy. We show that Quest can achieve up to 2.23x self-attention speedup, which reduces inference latency by 7.03x while performing well on tasks with long dependencies with negligible accuracy loss. Code is available at http://github.com/mit-han-lab/Quest .

Key facts. Cited by 487 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 25 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

11 of 25 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 10

  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  6. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  7. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  8. 2021Efficient Attentions for Long Document Summarizationcited by 342
  9. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  10. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 37; 2025: 163; 2026: 287.

487 papers.

Sort:

By area:

  1. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  2. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  3. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  4. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  5. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  6. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  7. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  8. 2025MemOS: A Memory OS for AI Systemcited by 129
  9. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  10. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  11. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  12. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
38 more
  1. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  2. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  3. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  4. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  5. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  6. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  7. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  8. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  9. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compressioncited by 65
  10. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
  11. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  12. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  13. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  14. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  15. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  16. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  17. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  18. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 45
  19. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  20. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 41
  21. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  22. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  23. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 38
  24. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  25. 2026MiniMax Sparse Attentioncited by 34
  26. 2025Training-Free Efficient Video Generation via Dynamic Token Carvingcited by 33
  27. 2025DrDiff: Dynamic Routing Diffusion with Hierarchical Attention for Breaking the Efficiency-Quality Trade-offcited by 33
  28. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  29. 2024TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attentioncited by 28
  30. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 28
  31. 2025Cognitive Memory in Large Language Modelscited by 28
  32. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  33. 2025Inference-Time Hyper-Scaling with KV Cache Compressioncited by 28
  34. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  35. 2025Tensor Product Attention Is All You Needcited by 27
  36. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  37. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
  38. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26

50 most cited of 487 All 487 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023SparQ Attention: Bandwidth-Efficient LLM Inference
  2. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
  3. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  4. 2024ThinK: Thinner Key Cache by Query-Driven Pruning
  5. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
  6. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  7. 2024In-context KV-Cache Eviction for LLMs via Attention-Gate
  8. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference