mlsanityData to 7 Oct 2026

Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters

arXiv 2406.12335 ↗First posted 18 Jun 2024cs.CL cs.LGcited by 73

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Scaling the context size of large language models (LLMs) enables them to perform various new tasks, e.g., book summarization. However, the memory cost of the Key and Value (KV) cache in attention significantly limits the practical applications of LLMs. Recent works have explored token pruning for KV cache reduction in LLMs, relying solely on attention scores as a token importance indicator. However, our investigation into value vector norms revealed a notably non-uniform pattern questioning their reliance only on attention scores. Inspired by this, we propose a new method: Value-Aware Token Pruning (VATP) which uses both attention scores and the ℓ1 norm of value vectors to evaluate token importance. Extensive experiments on LLaMA2-7B-chat and Vicuna-v1.5-7B across 16 LongBench tasks demonstrate that VATP outperforms attention-score-only baselines in over 12 tasks, confirming the effectiveness of incorporating value vector norms into token importance evaluation of LLMs.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 32 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

21 of 32 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  2. 2024On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference9 Feb
  3. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
5 more
  1. 2019Attention is not not Explanationcited by 629
  2. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  3. 2022Transkimmer: Transformer Learns to Layer-wise Skimcited by 21
  4. 2022How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformerscited by 20
  5. 2022Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERTcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 27; 2026: 35.

73 papers.

Sort:

By area:

  1. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  3. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  4. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
  5. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 38
  6. 2025Cognitive Memory in Large Language Modelscited by 28
  7. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  8. 2024Value Residual Learningcited by 25
  9. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  10. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 18
  11. 2025FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformerscited by 17
  12. 2025Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddingscited by 16
38 more
  1. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
  2. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 16
  3. 2025LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocationcited by 16
  4. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  5. 2025Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Querycited by 15
  6. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13
  7. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generationcited by 11
  8. 2025Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Coveringcited by 10
  9. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inferencecited by 9
  10. 2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothingcited by 9
  11. 2025Identifying and Evaluating Inactive Heads in Pretrained LLMscited by 8
  12. 2025Q-KVComm: Efficient Multi-Agent Communication Via Adaptive KV Cache Compressioncited by 8
  13. 2026Value-Aware Stochastic KV Cache Eviction for Reasoning Modelscited by 8
  14. 2025QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Designcited by 7
  15. 2024HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashingcited by 6
  16. 2025SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matchingcited by 6
  17. 2026LongFlow: Efficient KV Cache Compression for Reasoning Modelscited by 6
  18. 2024Cross-layer Attention Sharing for Pre-trained Large Language Modelscited by 5
  19. 2025Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsitycited by 5
  20. 2025GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Evictioncited by 5
  21. 2026On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compressioncited by 5
  22. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  23. 2025Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compressioncited by 4
  24. 2025Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearningcited by 4
  25. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMscited by 4
  26. 2026When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compressioncited by 4
  27. 2025Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMscited by 3
  28. 2025Quantifying surprise in clinical care: Detecting highly informative events in electronic health records with foundation modelscited by 3
  29. 2026DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inferencecited by 3
  30. 2024RecurFormer: Not All Transformer Heads Need Self-Attentioncited by 2
  31. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inferencecited by 2
  32. 2025WebRec: Enhancing LLM-based Recommendations with Attention-guided RAG from Webcited by 2
  33. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  34. 2025Energy-Efficient Wireless LLM Inference via Uncertainty and Importance-Aware Speculative Decodingcited by 1
  35. 2025Activation Quantization of Vision Encoders Needs Prefixing Registerscited by 1
  36. 2026CAPA: Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Modelscited by 1
  37. 2026Enhancing Multi-Image Understanding through Delimiter Token Scalingcited by 1
  38. 2026CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilationcited by 1

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  2. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  3. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
  4. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  5. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution
  6. 2026Fast KV Compaction via Attention Matching
  7. 2026Information-Aware KV Cache Compression for Long Reasoning
  8. 2025CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Eviction