mlsanityData to 7 Oct 2026

PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference

arXiv 2604.24971 ↗First posted 27 Apr 2026cs.LG cs.CL cs.DCcited by 3

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more
Abstract (arXiv)

We present PolyKV, a system in which multiple concurrent inference agents share a single, asymmetrically compressed KV cache pool. Rather than allocating a separate KV cache per agent -- the standard paradigm -- PolyKV writes a compressed cache once and injects it into N independent agent contexts via HuggingFace DynamicCache objects. Compression is asymmetric: Keys are quantized at int8 (q8_0) to preserve softmax stability, while Values are compressed using TurboQuant MSE -- a Fast Walsh-Hadamard Transform (FWHT) rotation followed by 3-bit Lloyd-Max quantization with centroids tuned to N(0,1). We evaluate across two model scales (SmolLM2-1.7B-Instruct and Llama-3-8B-Instruct), three context lengths (600-7,194 tokens), and up to 15 concurrent agents. PolyKV achieves a stable 2.91x compression ratio across all configurations. On Llama-3-8B with 15 agents sharing a 4K-token context, PolyKV reduces KV cache memory from 19.8 GB to 0.45 GB -- a 97.7% reduction -- while maintaining only +0.57% perplexity degradation and a mean BERTScore F1 of 0.928. PPL delta does not grow with agent count and improves as context length increases, inverting to -0.26% at 1,851 coherent tokens. To our knowledge, no prior work combines a single shared, lossy-compressed KV pool with multi-reader concurrent agent access.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 15 of its 25 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

15 of 25 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse28 Feb
  2. 2026Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices17 Feb
  3. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents1 Feb
  4. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems14 Oct
  5. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows10 Jul

Other references · 10

  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  3. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 148
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  5. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
  6. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurationscited by 19
  7. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 11
  8. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correctioncited by 9
  9. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 9
  10. 2026TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantizationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantizationcited by 0
  2. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inferencecited by 0
  3. 2026DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks
  2. 2026StepKV: Step-Aware KV Cache Compression for LLM Agents
  3. 2025Q-KVComm: Efficient Multi-Agent Communication Via Adaptive KV Cache Compression
  4. 2026ActKV: Efficient LLM Agents through Action-Guided KV Cache Management
  5. 2026Joint Encoding of KV-Cache Blocks for Scalable LLM Serving
  6. 2026SparseEngine: Sparse-First Inference Engine
  7. 2026AnchorKV: Anchor-Residual KV Cache Compression
  8. 2026MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference