# Attention-Weighted Value Projection for KV-Cache Compression

arXiv: https://arxiv.org/abs/2604.11501 · HTML: https://mlsanity.com/paper/2604.11501/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 4 of its 5 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 4 of 4)

- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration](https://mlsanity.com/paper/2505.24357/) (2025)
- [KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity](https://mlsanity.com/paper/2512.05916/) (2025)
- [CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios](https://mlsanity.com/paper/2409.10593/) (2024)

## Built on by (most cited first, 2 of 2)

- [Which Decisions Low-Bit Quantization Breaks, and How to Predict Them](https://mlsanity.com/paper/2608.06564/) (2026)
- [Variance Is Not Importance: Structural Analysis of Transformer Compressibility Across Model Scales](https://mlsanity.com/paper/2604.20682/) (2026)

## Abstract (arXiv)

Rank reduction discards dimensions; quantization keeps them at lower precision. Comparing the two requires a choice of what compression should preserve. For attention values, we study reconstruction of the attention output rather than reconstruction of the values alone. With fixed attention weights, the optimal orthogonal rank-r projection uses the leading eigenvectors of V^topα^topα V, and its error is exactly the discarded eigenvalue sum. We extend this objective to calibration datasets and grouped query attention, and describe rank allocation under an additive local error budget. We also examine the limits of using local error to predict downstream loss. Historical weight-perturbation experiments favor coefficient rounding over the value projections tested, but do not establish a comparison at equal cache storage. The resulting distinction is practical: the projection objective has an exact solution, while a comparison with cache quantization requires separate activation-level experiments.