mlsanityData to 14 Oct 2026

AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models

arXiv 2506.03762 ↗First posted 4 Jun 2025cs.CL cs.AIcited by 4

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models (LLMs) have significantly advanced the field of Artificial Intelligence. However, their deployment is resource-intensive, not only due to the large number of model parameters but also because the (Key-Value) KV cache consumes a lot of memory during inference. While several works propose reducing the KV cache by evicting the unnecessary tokens, these approaches rely on accumulated attention score as eviction score to quantify the importance of the token. We identify the accumulated attention score is biased and it decreases with the position of the tokens in the mathematical expectation. As a result, the retained tokens concentrate on the initial positions, limiting model's access to global contextual information. To address this issue, we propose Adaptive holistic attention KV (AhaKV), it addresses the bias of the accumulated attention score by adaptively tuning the scale of softmax according the expectation of information entropy of attention scores. To make use of the holistic attention information in self-attention mechanism, AhaKV utilize the information of value vectors, which is overlooked in previous works, to refine the adaptive score. We show theoretically that our method is well suited for bias reduction. We deployed AhaKV on different models with a fixed cache budget. Experiments show that AhaKV successfully mitigates bias and retains crucial tokens across global context and achieve state-of-the-art results against other related work on several benchmark tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 42 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

26 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time7 Aug
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  3. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2020Longformer: The Long-Document Transformercited by 3,413
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
10 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  8. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  9. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  10. 2023Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddingscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compressioncited by 5
  2. 2026Epiphany-Aware KV Cache Eviction Without the Attention Matrixcited by 2
  3. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inferencecited by 1
  4. 2026ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inferencecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024In-context KV-Cache Eviction for LLMs via Attention-Gate
  2. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  3. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  4. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
  5. 2026CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
  6. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
  7. 2025MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference
  8. 2024Beyond KV Caching: Shared Attention for Efficient LLMs