mlsanityData to 7 Oct 2026

Dynamic HBM Repartitioning for Multi-Turn MoE Serving

arXiv 2609.13537 ↗First posted 11 Sep 2026cs.DCcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Long-running multi-turn requests accumulate reusable key-value (KV) state. Once this state exceeds a fixed GPU KV-cache allocation, serving systems evict reusable prefixes, repeat prefill work, and may preempt requests. This pressure is particularly acute for Mixture-of-Experts (MoE) models: their expert weights occupy most GPU high-bandwidth memory (HBM), even though each token activates only a sparse subset of experts. A static boundary between weights and the KV cache prevents serving systems from using expert memory to preserve reusable state as conversations grow. We present VAMP, an MoE serving framework that changes this HBM boundary at runtime. When a KV allocation cannot be satisfied, VAMP compares the estimated future work of three alternatives: staging expert weights from host memory, evicting cached prefixes that may require re-prefill, or preempting and rescheduling requests. It then converts a bounded expert-weight region into KV-cache capacity when that action has the lowest estimated penalty. CUDA Virtual Memory Management page remapping performs this conversion without copying resident KV data. We implement VAMP in the vLLM serving engine and evaluate Qwen3-Next-80B on recorded and controlled multi-turn workloads. Across five replays of a recorded 2,103-turn SWE-bench agent workload, VAMP with a 15% maximum expert-offloading ratio reduces time-to-first-token (TTFT) p90 from 26.1 s to 1.10 s (23.6 times) and increases request throughput by 20.7% relative to unmodified vLLM, while increasing time per output token (TPOT) by 31.1%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 24 of its 30 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

24 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation23 Jun
  2. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  3. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb

Other references · 21

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2024Mixtral of Expertscited by 1,785
  5. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  8. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  9. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  10. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  11. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  12. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
9 more
  1. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  2. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  3. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
  4. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  5. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 23
  6. 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inferencecited by 15
  7. 2024GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitchingcited by 13
  8. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 13
  9. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switch
  2. 2026ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels
  3. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
  4. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  5. 2026Beyond Capacity: Scalable MoE LLM Inference via High-Bandwidth Flash with Direct GPU and HBM Paths
  6. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution
  7. 2026FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
  8. 2025QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration