mlsanityData to 14 Oct 2026

Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval

arXiv 2508.19740 ↗First posted 27 Aug 2025cs.CLcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Reducing the key-value (KV) cache burden in Large Language Models (LLMs) significantly accelerates inference. Dynamically selecting critical KV caches during decoding helps maintain performance. Existing methods use random linear hashing to identify important tokens, but this approach is inefficient due to the orthogonal distribution of queries and keys within two narrow cones in LLMs. We introduce Spotlight Attention, a novel method that employs non-linear hashing functions to optimize the embedding distribution of queries and keys, enhancing coding efficiency and robustness. We also developed a lightweight, stable training framework using a Bradley-Terry ranking-based loss, enabling optimization of the non-linear hashing module on GPUs with 16GB memory in 8 hours. Experimental results show that Spotlight Attention drastically improves retrieval precision while shortening the length of the hash code at least 5× compared to traditional linear hashing. Finally, we exploit the computational advantages of bitwise operations by implementing specialized CUDA kernels, achieving hashing retrieval for 512K tokens in under 100μs on a single A100 GPU, with end-to-end throughput up to 3× higher than vanilla decoding.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 25 references that are in the map. First posted on arXiv 27 Aug 2025.

Builds on

20 of 25 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  2. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time7 Aug
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  7. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  8. 2024Transformers are Multi-State RNNscited by 120
  9. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  10. 2015Practical and Optimal LSH for Angular Distancecited by 106
  11. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  12. 2025UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Modelscited by 11
1 more
  1. 2024Low-Rank Interconnected Adaptation across Layerscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Training-Free Hashing-Based Attention via Binary Principal Componentscited by 0
  2. 2026HHR: Hierarchical Hash Retrieval for Efficient LLM Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing
  2. 2026DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing
  3. 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs
  4. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
  5. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs
  6. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  7. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  8. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference