# PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference

arXiv: https://arxiv.org/abs/2604.24971 · HTML: https://mlsanity.com/paper/2604.24971/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)
- Efficient LLM inference and serving / [Efficient reasoning and agent inference](https://mlsanity.com/inference/efficient-reasoning-and-agent-inference/)

## Builds on (most cited first, 15 of 15)

- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference](https://mlsanity.com/paper/2405.12532/) (2024)
- [LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference](https://mlsanity.com/paper/2510.09665/) (2025)
- [KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows](https://mlsanity.com/paper/2507.07400/) (2025)
- [RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations](https://mlsanity.com/paper/2501.16383/) (2025)
- [KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems](https://mlsanity.com/paper/2510.12872/) (2025)
- [AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations](https://mlsanity.com/paper/2410.13212/) (2024)
- [KV Cache Transform Coding for Compact Storage in LLM Inference](https://mlsanity.com/paper/2511.01815/) (2025)
- [KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction](https://mlsanity.com/paper/2510.05373/) (2025)
- [EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving](https://mlsanity.com/paper/2512.14946/) (2025)
- [LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents](https://mlsanity.com/paper/2602.01053/) (2026)
- [Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices](https://mlsanity.com/paper/2603.04428/) (2026)
- [RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse](https://mlsanity.com/paper/2603.13289/) (2026)
- [TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantization](https://mlsanity.com/paper/2603.27467/) (2026)

## Built on by (most cited first, 3 of 3)

- [DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression](https://mlsanity.com/paper/2607.06523/) (2026)
- [HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization](https://mlsanity.com/paper/2605.03562/) (2026)
- [FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference](https://mlsanity.com/paper/2607.06519/) (2026)

## Abstract (arXiv)

We present PolyKV, a system in which multiple concurrent inference agents share a single, asymmetrically compressed KV cache pool. Rather than allocating a separate KV cache per agent -- the standard paradigm -- PolyKV writes a compressed cache once and injects it into N independent agent contexts via HuggingFace DynamicCache objects. Compression is asymmetric: Keys are quantized at int8 (q8_0) to preserve softmax stability, while Values are compressed using TurboQuant MSE -- a Fast Walsh-Hadamard Transform (FWHT) rotation followed by 3-bit Lloyd-Max quantization with centroids tuned to N(0,1). We evaluate across two model scales (SmolLM2-1.7B-Instruct and Llama-3-8B-Instruct), three context lengths (600-7,194 tokens), and up to 15 concurrent agents. PolyKV achieves a stable 2.91x compression ratio across all configurations. On Llama-3-8B with 15 agents sharing a 4K-token context, PolyKV reduces KV cache memory from 19.8 GB to 0.45 GB -- a 97.7% reduction -- while maintaining only +0.57% perplexity degradation and a mean BERTScore F1 of 0.928. PPL delta does not grow with agent count and improves as context length increases, inverting to -0.26% at 1,851 coherent tokens. To our knowledge, no prior work combines a single shared, lossy-compressed KV pool with multi-reader concurrent agent access.