mlsanityData to 7 Oct 2026

Recall Before You Rank: Similarity-Guided Top-K Reuse for Efficient Long-Context Attention

arXiv 2607.27692 ↗First posted 30 Jul 2026cs.CL cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Top-K sparse attention reduces the cost of Softmax and value aggregation by attending to only a small subset of key--value (KV) entries. However, identifying this subset still requires scoring the current query against the full KV cache and performing global Top-K selection, leaving selector cost linear in context length and limiting the practical efficiency of sparse attention for long-context decoding. In this paper, we introduce ReTopK, a training-free method that accelerates dynamic Top-K attention by reusing historical retrieval decisions. ReTopK builds on the observation that similar queries often attend to overlapping supports and that partially overlapping supports can still preserve most of the Exact Top-K attention mass. For each attention head, it maintains a bounded cache of historical query--support pairs, retrieves the most similar cached queries for each new query, unions their stored supports with a recent window, and reranks only the resulting compact candidate set using exact current-query scores. A similarity-based fallback invokes full-history Exact Top-K when reuse is unreliable, while periodic exact refreshes limit cache drift. ReTopK retains the complete KV cache and reuses only selected indices, rather than historical scores, attention weights, or outputs. Across 16K--128K contexts, ReTopK achieves the lowest PG19 perplexity and the highest NIAH and LongBench scores among the evaluated approximate methods. At 128K with K=512, ReTopK incurs only a 0.50% perplexity increase over Exact Top-K while accelerating attention computation by 3.07×.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 22 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

21 of 22 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection5 Nov
  2. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  6. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 14

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  8. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  9. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  10. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  11. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  12. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
2 more
  1. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inferencecited by 9
  2. 2026Predict, Reuse, and Repair: Accelerating Dynamic Sparse Attention for Long-Context LLM Decodingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
  2. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs
  3. 2026HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads
  4. 2025Retrospective Sparse Attention for Efficient Long-Context Generation
  5. 2026You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
  6. 2026OVAL: Output-Aware Local Page Bases for KV Cache Retrieval
  7. 2026Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
  8. 2026UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training