mlsanityData to 7 Oct 2026

Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query

arXiv 2505.20334 ↗First posted 24 May 2025cs.CL cs.AIcited by 15

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) rely on key-value cache (KV cache) to accelerate decoding by reducing redundant computations. However, the KV cache memory usage grows substantially with longer text sequences, posing challenges for efficient deployment. Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries, especially under tight memory budgets. In this paper, we propose Lookahead Q-Cache (LAQ), a novel eviction framework that generates low-cost pseudo lookahead queries to better approximate the true decoding-stage queries. By using these lookahead queries as the observation window for importance estimation, LAQ achieves more consistent and accurate KV cache eviction aligned with real inference scenarios. Experimental results on LongBench and Needle-in-a-Haystack benchmarks show that LAQ outperforms existing methods across various budget levels, achieving a 1 ~ 4 point improvement on LongBench under limited cache budget. Moreover, LAQ is complementary to existing approaches and can be flexibly combined to yield further improvements.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 30 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

22 of 30 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  2. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity3 Dec
  3. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  4. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches1 Jul
  5. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters18 Jun
  6. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  10. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 12

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  8. 2021Efficient Attentions for Long Document Summarizationcited by 342
  9. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  10. 2025A Comprehensive Survey on Long Context Language Modelingcited by 113
  11. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  12. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 13.

15 papers.

Sort:

By area:

  1. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generationcited by 11
  2. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inferencecited by 7
  3. 2025Draft-based Approximate Inference for LLMscited by 6
  4. 2025Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Evictioncited by 4
  5. 2026When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compressioncited by 4
  6. 2026EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstructioncited by 2
  7. 2026KV2: A Self-Refining KV Cachecited by 1
  8. 2026Monte Carlo Estimation for KV Cache Evictioncited by 1
  9. 2026KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Mergingcited by 0
  10. 2026Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compressioncited by 0
  11. 2026Discovering KV Cache Eviction Policies via LLM-Guided Program Evolutioncited by 0
  12. 2026When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compressioncited by 0
3 more
  1. 2026DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compressioncited by 0
  2. 2026Mask-Guided KV Cache Eviction in Block Diffusion Language Modelscited by 0
  3. 2026Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Readcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Where Matters More Than What: Decoding-aligned KV Cache Compression via Position-aware Pseudo Queries
  2. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  3. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time
  4. 2024ThinK: Thinner Key Cache by Query-Driven Pruning
  5. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
  6. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
  7. 2026Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective
  8. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective