# Trust the Mass: Forced Weights in KV-Cache Eviction

arXiv: https://arxiv.org/abs/2608.25230 · HTML: https://mlsanity.com/paper/2608.25230/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 25 Aug 2026.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 26 of 26)

- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024)
- [DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads](https://mlsanity.com/paper/2410.10819/) (2024)
- [Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference](https://mlsanity.com/paper/2407.11550/) (2024)
- [Landmark Attention: Random-Access Infinite Context Length for Transformers](https://mlsanity.com/paper/2305.16300/) (2023)
- [Transformers are Multi-State RNNs](https://mlsanity.com/paper/2401.06104/) (2024)
- [KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction](https://mlsanity.com/paper/2505.23416/) (2025)
- [Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution](https://mlsanity.com/paper/2510.00636/) (2025)
- [Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters](https://mlsanity.com/paper/2406.12335/) (2024)
- [Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruning](https://mlsanity.com/paper/2502.02770/) (2025)
- [The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks](https://mlsanity.com/paper/2603.05498/) (2026)
- [CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective](https://mlsanity.com/paper/2502.03805/) (2025)
- [Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores](https://mlsanity.com/paper/2507.08143/) (2025)
- [Streaming Attention Approximation via Discrepancy Theory](https://mlsanity.com/paper/2502.07861/) (2025)
- [Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction](https://mlsanity.com/paper/2605.09649/) (2026)
- [CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation](https://mlsanity.com/paper/2508.02401/) (2025)
- [IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference](https://mlsanity.com/paper/2605.25475/) (2026)
- [SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space](https://mlsanity.com/paper/2511.20102/) (2025)
- [When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression](https://mlsanity.com/paper/2605.08234/) (2026)
- [A Mathematical Theory of Top-k Sparse Attention via Total Variation Distance](https://mlsanity.com/paper/2512.07647/) (2025)
- [LevAttention: Time, Space, and Streaming Efficient Algorithm for Heavy Attentions](https://mlsanity.com/paper/2410.05462/) (2024)
- [Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving](https://mlsanity.com/paper/2606.06302/) (2026)
- [How Much Dense Attention is Necessary? Oracle-Guided Sparse Prefill for Full/GQA Layers in Hybrid Long-Context Models](https://mlsanity.com/paper/2606.07703/) (2026)

## Built on by (most cited first, 1 of 1)

- [CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache](https://mlsanity.com/paper/2610.02235/) (2026)

## Abstract (arXiv)

Every deployed sparse-attention or KV-cache-eviction rule keeps a subset of the keys, discards the rest, and renormalizes the attention weights over the kept set. Enumerating the exact best subset under that constraint on 168,192 attention rows from five models shows that keeping the largest weights is already near-optimal, since the best subset closes only a median 2 to 5% of the remaining gap to full attention. If selection closes this little, published margins between eviction methods must come from elsewhere, so we measure the bytes each method holds. In the shared evaluation pipeline, the strongest query-agnostic methods hold the full cache because their per-head selections are stored as masks, and only ragged per-head storage frees that memory. Enforcing a nominal budget on one fixed selection costs 14 to 62 benchmark points. We trace an 87.6-point retrieval margin to rankings computed while the question is visible. ContourKV, a training-free allocator built from the dropped-mass statistic, wins 93 of 160 paired comparisons against that state of the art and loses 22 at the byte count of the budget-enforcing baselines, and it ties the strongest of them.