# PiKV: KV Cache Management System for Mixture of Experts

arXiv: https://arxiv.org/abs/2508.06526 · HTML: https://mlsanity.com/paper/2508.06526/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 2 Aug 2025.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)
- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 30 of 30)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024)
- [Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention](https://mlsanity.com/paper/2403.19708/) (2024)
- [Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models](https://mlsanity.com/paper/2402.14800/) (2024)
- [Training-Free Long-Context Scaling of Large Language Models](https://mlsanity.com/paper/2402.17463/) (2024)
- [PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformation](https://mlsanity.com/paper/2301.10936/) (2023)
- [FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation](https://mlsanity.com/paper/2505.20353/) (2025)
- [HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing](https://mlsanity.com/paper/2405.06067/) (2024)
- [MT2ST: Adaptive Multi-Task to Single-Task Learning](https://mlsanity.com/paper/2406.18038/) (2024)
- [LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference](https://mlsanity.com/paper/2406.19657/) (2024)
- [Designing Large Foundation Models for Efficient Training and Inference: A Survey](https://mlsanity.com/paper/2409.01990/) (2024)

## Built on by (most cited first, 5 of 5)

- [TinyServe: Query-Aware Cache Selection for Efficient LLM Serving](https://mlsanity.com/paper/2509.12211/) (2025)
- [CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inference](https://mlsanity.com/paper/2511.21702/) (2025)
- [QuickMerge++: Fast Token Merging with Autoregressive Prior](https://mlsanity.com/paper/2508.13204/) (2025)
- [Cognitive Load Traces as Symbolic and Visual Accounts of Deep Model Cognition](https://mlsanity.com/paper/2510.15980/) (2025)
- [HSMLA: Hierarchical Softmax Multi-scale Linear Attention for Efficient Vision Transformers](https://mlsanity.com/paper/2608.07616/) (2026)

## Abstract (arXiv)

As large-scale language models continue to scale up in both size and context length, the memory and communication cost of key-value (KV) cache storage has become a major bottleneck in multi-GPU and multi-node inference. While MoE-based architectures sparsify computation across experts, the corresponding KV caches remain dense and globally synchronized, resulting in significant overhead. We introduce PiKV, a parallel and distributed KV cache serving framework tailored for MoE architecture. PiKV leverages textitexpert-sharded KV storage to partition caches across GPUs, textitPiKV routing to reduce token-to-KV access, and a textitPiKV Scheduling to adaptively retain query-relevant entries. To further reduce memory usage, PiKV integrates textitPiKV Compression modules the caching pipeline for acceleration. PiKV is recently publicly available as an open-source software library: hrefhttps://github.com/NoakLiu/PiKVhttps://github.com/NoakLiu/PiKV. PiKV is still a living project, aiming to become a comprehesive KV Cache management system for MoE Architectures.