mlsanityData to 14 Oct 2026

NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time

arXiv 2408.03675 ↗First posted 7 Aug 2024cs.CLcited by 43

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models (LLMs) have ignited an innovative surge of AI applications, marking a new era of exciting possibilities equipped with extended context windows. However, hosting these models is cost-prohibitive mainly due to the extensive memory consumption of KV Cache involving long-context modeling. Despite several works proposing to evict unnecessary tokens from the KV Cache, most of them rely on the biased local statistics of accumulated attention scores and report performance using unconvincing metric like perplexity on inadequate short-text evaluation. In this paper, we propose NACL, a general framework for long-context KV cache eviction that achieves more optimal and efficient eviction in a single operation during the encoding phase. Due to NACL's efficiency, we combine more accurate attention score statistics in PROXY TOKENS EVICTION with the diversified random eviction strategy of RANDOM EVICTION, aiming to alleviate the issue of attention bias and enhance the robustness in maintaining pivotal tokens for long-context modeling tasks. Notably, our method significantly improves the performance on short- and long-text tasks by 80% and 76% respectively, reducing KV Cache by up to 50% with over 95% performance maintenance. The code is available at https://github.com/PaddlePaddle/Research/tree/master/NLP/ACL2024-NACL.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 45 references that are in the map. First posted on arXiv 7 Aug 2024.

Builds on

29 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
15 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  2. 2020Reformer: The Efficient Transformercited by 1,668
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2020Efficient Transformers: A Surveycited by 790
  7. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  8. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  10. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  11. 2023Adapting Language Models to Compress Contextscited by 356
  12. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  13. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  14. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  15. 2020ERNIE-Doc: A Retrospective Long-Document Modeling Transformercited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 16; 2026: 19.

43 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  2. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 95
  3. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  4. 2024FlashMask: Efficient and Rich Mask Extension of FlashAttentioncited by 20
  5. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 19
  6. 2025LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoningcited by 16
  7. 2024In-context KV-Cache Eviction for LLMs via Attention-Gatecited by 13
  8. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  9. 2024The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trendscited by 11
  10. 2024RefreshKV: Updating Small KV Cache During Long-form Generationcited by 11
  11. 2025LazyMAR: Accelerating Masked Autoregressive Models via Feature Cachingcited by 11
  12. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?cited by 10
31 more
  1. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generationcited by 10
  2. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 9
  3. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 8
  4. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  5. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 6
  6. 2026KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasingcited by 6
  7. 2024Inference-Friendly Models With MixAttentioncited by 4
  8. 2024EMS: Adaptive Evict-then-Merge Strategy for Head-wise KV Cache Compression Based on Global-Local Importancecited by 4
  9. 2025AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Modelscited by 4
  10. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limitscited by 4
  11. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 4
  12. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cachecited by 4
  13. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  14. 2025Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Evictioncited by 3
  15. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inferencecited by 3
  16. 2026MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inferencecited by 3
  17. 2025Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrievalcited by 2
  18. 2025Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understandingcited by 2
  19. 2026HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compressioncited by 2
  20. 2026Prefix Sliding for efficient test-time scalingcited by 2
  21. 2026Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stabilitycited by 1
  22. 2026Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turnscited by 1
  23. 2026Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometrycited by 1
  24. 2026Random Attention: Rethinking KV Cache Eviction for Efficient Reasoningcited by 1
  25. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstructioncited by 1
  26. 2025AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMscited by 0
  27. 2026Hierarchical Adaptive Eviction for KV Cache Management in Multimodal Language Modelscited by 0
  28. 2026S4R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Cachingcited by 0
  29. 2026GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Servingcited by 0
  30. 2026Understanding and Exploiting Diagonal Attention Sparsity in Autoregressive Image Generationcited by 0
  31. 2026AvoKV-E: Payload-Aware KV Cache Eviction for Long Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  2. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  3. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attention
  4. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
  5. 2026Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
  6. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
  8. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences