mlsanityData to 7 Oct 2026

Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression

arXiv 2503.02812 ↗First posted 4 Mar 2025cs.CL cs.AIcited by 4

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Autoregressive language models rely on a Key-Value (KV) Cache, which avoids re-computing past hidden states during generation, making it faster. As model sizes and context lengths grow, the KV Cache becomes a significant memory bottleneck, which calls for compression methods that limit its size during generation. In this paper, we discover surprising properties of Query (Q) and Key (K) vectors that allow us to efficiently approximate attention scores without computing the attention maps. We propose Q-Filters, a training-free KV Cache compression method that filters out less crucial Key-Value pairs based on a single context-agnostic projection. Contrarily to many alternatives, Q-Filters is compatible with FlashAttention, as it does not require direct access to attention weights. Experimental results in long-context settings demonstrate that Q-Filters is competitive with attention-based compression methods such as SnapKV in retrieval tasks while consistently outperforming efficient compression schemes such as Streaming-LLM in generation setups. Notably, Q-Filters achieves a 99% accuracy in the needle-in-a-haystack task with a x32 compression level while reducing the generation perplexity drop by up to 65% in text generation compared to Streaming-LLM.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 22 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

18 of 22 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters18 Jun
  2. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  3. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  10. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  11. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  12. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 62
1 more
  1. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  2. 2025The Pitfalls of KV Cache Compressioncited by 15
  3. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
  4. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstructioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026KV2: A Self-Refining KV Cache
  2. 2025CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
  3. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity
  4. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning
  5. 2026A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
  6. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
  7. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
  8. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques