mlsanityData to 14 Oct 2026

Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference

arXiv 2609.14507 ↗First posted 13 Sep 2026cs.DC cs.LGcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the key-value cache (KVCache) to CPU memory. The spilled KV serves two complementary purposes---transferring to the GPU for attention computation, or computing in-place on the CPU---which demand opposing physical states. The optimal split between them varies with workload, yet existing KVCache abstractions offer only storage semantics over a monolithic object of a single physical state, and cannot express dynamic load balancing. We propose InplaceKVCache, the first KVCache abstraction whose format fixes each byte's physical residency at write time, so that the CPU--GPU load balance can be adjusted without moving data after placement. It realizes this as a four-region layout along two dimensions---device affinity and access pattern---turning load balancing into pure scheduling. Built on this abstraction, WriteScope splits CPU--GPU shares along the sequence dimension, and a portable roofline performance model determines the optimal CPU share as sequence length evolves, with online feedback tracking CPU cost drift. On three MoE models (DeepSeek-V2-Lite, Qwen3-30B-A3B, Mixtral-8×7B) with a 32~GB VRAM budget, WriteScope supports end-to-end inference at the 1M-token aggregate scale. In the long-context regime (≥8K), it achieves geometric-mean speedups of 1.5×--2.5× on A100 and 1.4×--1.7× on V100 over four reproduced baselines, while vLLM, SGLang, and KTransformers fail even with a doubled KV budget. A DeepSeek-V4-Flash case study validates composition with native sparse attention.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 28 of its 49 references that are in the map. First posted on arXiv 13 Sep 2026.

Builds on

28 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inference15 Dec
  2. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers28 Sep
  3. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference8 Apr
  4. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading7 Feb
  5. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  6. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb

Other references · 22

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  8. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  9. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  10. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  11. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  12. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
10 more
  1. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  2. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 126
  3. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  4. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  5. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  6. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloadingcited by 9
  7. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inferencecited by 5
  8. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inferencecited by 3
  9. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computingcited by 2
  10. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inferencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025FairKV: Balancing Per-Head KV Cache for Fast Multi-GPU Inference
  2. 2026Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
  3. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
  4. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
  5. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  6. 2025Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems
  7. 2025PiKV: KV Cache Management System for Mixture of Experts
  8. 2025Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference