# WeightedKV: Attention Scores Weighted Key-Value Cache Merging for Large Language Models

arXiv: https://arxiv.org/abs/2503.01330 · HTML: https://mlsanity.com/paper/2503.01330/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 16 of its 19 references that are in the map.
- First posted on arXiv 3 Mar 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time](https://mlsanity.com/paper/2305.17118/) (2023)
- [LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models](https://mlsanity.com/paper/2308.16137/) (2023)
- [Transformers are Multi-State RNNs](https://mlsanity.com/paper/2401.06104/) (2024)
- [Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference](https://mlsanity.com/paper/2403.09636/) (2024)
- [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://mlsanity.com/paper/2305.06984/) (2023)
- [Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks](https://mlsanity.com/paper/2407.08454/) (2024)
- [Anchor-based Large Language Models](https://mlsanity.com/paper/2402.07616/) (2024)

## Built on by (most cited first, 4 of 4)

- [ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference](https://mlsanity.com/paper/2602.08005/) (2026)
- [SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Merging](https://mlsanity.com/paper/2603.14303/) (2026)
- [SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation](https://mlsanity.com/paper/2607.16213/) (2026)
- [Beyond Sparse Weights: When Is Attention Compressible?](https://mlsanity.com/paper/2608.21541/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) use key-value (KV) cache to reduce redundant computation in autoregressive generation. However, the KV cache size increases linearly during generation, leading to excessive memory usage, especially for long texts. Most KV cache compression methods evict the unimportant KV pairs to maintain a fixed cache size, which leads to the permanent loss of tokens during generation. However, singular value decomposition shows that textitvalues do not exhibit a strong low-rank property as textitkeys do, suggesting that information is distributed more evenly across textitvalues, in contrast to its more redundant distribution within textitkeys. Therefore, methods that evict both textitkeys and textitvalues risk losing crucial information and compromise context integrity, ultimately degrading the output quality. To address this problem, we propose WeightedKV, a novel, training-free approach that discards the textitkeys of less important tokens, while merging their textitvalues into neighboring tokens via a convex combination weighted by their average attention scores. In this way, the retained textitkeys serve as anchors that guide the generation process, while the merged textitvalues provide a rich contextual backdrop. We assess our method on four widely used language modeling datasets, demonstrating superior performance compared to all baseline methods, particularly with a lower budget ratio.