mlsanityData to 7 Oct 2026

In-context KV-Cache Eviction for LLMs via Attention-Gate

arXiv 2410.12876 ↗First posted 15 Oct 2024cs.CL cs.LGcited by 12

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The KV-Cache technique has become the standard for the inference of large language models (LLMs). Yet, it is widely criticized that KV-Cache can become a bottleneck of the LLM inference system. This paper enables a novel dynamic KV-Cache eviction policy by injecting a lightweight module called Attention-Gate to the model. It accepts the global context as input and yields eviction flags for each token. The self-attention modules in the model proceed according to the flags and cache only a subset of the KV states for next token prediction. The Attention-Gates can yield various flags for different heads and layers and be easily tuned on top of a pre-trained LLM via continual pre-training or supervised fine-tuning. The computational and memory overhead introduced by Attention-Gates can be minimal. We empirically evaluate the proposed approach across multiple scenarios, showing that effective eviction of redundant tokens can not only improve efficiency but also enhance performance.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 36 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

27 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time7 Aug
  2. 2024A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decoder30 Jul
  3. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2023Mistral 7Bcited by 3,065
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
11 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  8. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  9. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
  10. 2024Transformers are Multi-State RNNscited by 120
  11. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 4.

12 papers.

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  3. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 18
  4. 2025AttentionPredictor: Temporal Patterns Matter for KV Cache Compressioncited by 7
  5. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 5
  6. 2025From Masks to Worlds: A Hitchhiker's Guide to World Modelscited by 5
  7. 2025Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Evictioncited by 3
  8. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
  9. 2026TABES: Trajectory-Aware Backward-on-Entropy Steering for Masked Diffusion Modelscited by 3
  10. 2026Language Models Can Control Their Own Attentioncited by 1
  11. 2026ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compactioncited by 1
  12. 2026Sigmoid Attention as a Better Substrate for Learned KV Cache Evictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  2. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  3. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
  4. 2026Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
  5. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  6. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attention
  7. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  8. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget