# Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query

arXiv: https://arxiv.org/abs/2505.20334 · HTML: https://mlsanity.com/paper/2505.20334/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 24 May 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 22 of 22)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time](https://mlsanity.com/paper/2305.17118/) (2023)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference](https://mlsanity.com/paper/2407.11550/) (2024)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://mlsanity.com/paper/2405.05254/) (2024)
- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [A Comprehensive Survey on Long Context Language Modeling](https://mlsanity.com/paper/2503.17407/) (2025)
- [ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification](https://mlsanity.com/paper/2405.14256/) (2024)
- [A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression](https://mlsanity.com/paper/2406.11430/) (2024)
- [Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters](https://mlsanity.com/paper/2406.12335/) (2024)
- [KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches](https://mlsanity.com/paper/2407.01527/) (2024)
- [A Survey on Transformer Context Extension: Approaches and Evaluation](https://mlsanity.com/paper/2503.13299/) (2025)
- [Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity](https://mlsanity.com/paper/2412.02252/) (2024)

## Built on by (most cited first, 14 of 14)

- [LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation](https://mlsanity.com/paper/2603.10899/) (2026)
- [IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference](https://mlsanity.com/paper/2605.25475/) (2026)
- [Draft-based Approximate Inference for LLMs](https://mlsanity.com/paper/2506.08373/) (2025)
- [When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression](https://mlsanity.com/paper/2605.08234/) (2026)
- [Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction](https://mlsanity.com/paper/2509.10798/) (2025)
- [EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction](https://mlsanity.com/paper/2603.22910/) (2026)
- [KV2: A Self-Refining KV Cache](https://mlsanity.com/paper/2610.03198/) (2026)
- [KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Merging](https://mlsanity.com/paper/2603.00907/) (2026)
- [Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compression](https://mlsanity.com/paper/2605.22337/) (2026)
- [Discovering KV Cache Eviction Policies via LLM-Guided Program Evolution](https://mlsanity.com/paper/2608.14555/) (2026)
- [When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compression](https://mlsanity.com/paper/2609.33334/) (2026)
- [Budgeted Cache Repair for Cross-Context KV-Cache Reuse](https://mlsanity.com/paper/2610.02233/) (2026)
- [Mask-Guided KV Cache Eviction in Block Diffusion Language Models](https://mlsanity.com/paper/2610.06996/) (2026)
- [Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Read](https://mlsanity.com/paper/2610.12133/) (2026)

## Abstract (arXiv)

Large language models (LLMs) rely on key-value cache (KV cache) to accelerate decoding by reducing redundant computations. However, the KV cache memory usage grows substantially with longer text sequences, posing challenges for efficient deployment. Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries, especially under tight memory budgets. In this paper, we propose Lookahead Q-Cache (LAQ), a novel eviction framework that generates low-cost pseudo lookahead queries to better approximate the true decoding-stage queries. By using these lookahead queries as the observation window for importance estimation, LAQ achieves more consistent and accurate KV cache eviction aligned with real inference scenarios. Experimental results on LongBench and Needle-in-a-Haystack benchmarks show that LAQ outperforms existing methods across various budget levels, achieving a 1 ~ 4 point improvement on LongBench under limited cache budget. Moreover, LAQ is complementary to existing approaches and can be flexibly combined to yield further improvements.