mlsanityData to 14 Oct 2026

PiKV: KV Cache Management System for Mixture of Experts

arXiv 2508.06526 ↗First posted 2 Aug 2025cs.DC cs.AI cs.ARcited by 5

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

As large-scale language models continue to scale up in both size and context length, the memory and communication cost of key-value (KV) cache storage has become a major bottleneck in multi-GPU and multi-node inference. While MoE-based architectures sparsify computation across experts, the corresponding KV caches remain dense and globally synchronized, resulting in significant overhead. We introduce PiKV, a parallel and distributed KV cache serving framework tailored for MoE architecture. PiKV leverages textitexpert-sharded KV storage to partition caches across GPUs, textitPiKV routing to reduce token-to-KV access, and a textitPiKV Scheduling to adaptively retain query-relevant entries. To further reduce memory usage, PiKV integrates textitPiKV Compression modules the caching pipeline for acceleration. PiKV is recently publicly available as an open-source software library: hrefhttps://github.com/NoakLiu/PiKVhttps://github.com/NoakLiu/PiKV. PiKV is still a living project, aiming to become a comprehesive KV Cache management system for MoE Architectures.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 30 of its 37 references that are in the map. First posted on arXiv 2 Aug 2025.

Builds on

30 of 37 references.

References · 30

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
18 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  4. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  5. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  8. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  10. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  11. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  12. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  13. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
  14. 2025FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximationcited by 24
  15. 2024HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processingcited by 21
  16. 2024MT2ST: Adaptive Multi-Task to Single-Task Learningcited by 16
  17. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inferencecited by 15
  18. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025TinyServe: Query-Aware Cache Selection for Efficient LLM Servingcited by 9
  2. 2025CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inferencecited by 7
  3. 2025QuickMerge++: Fast Token Merging with Autoregressive Priorcited by 1
  4. 2025Cognitive Load Traces as Symbolic and Visual Accounts of Deep Model Cognitioncited by 0
  5. 2026HSMLA: Hierarchical Softmax Multi-scale Linear Attention for Efficient Vision Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences
  2. 2026Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference
  3. 2026Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  5. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  6. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  7. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
  8. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference