mlsanityData to 7 Oct 2026

Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens

arXiv 2512.09277 ↗First posted 10 Dec 2025cs.DC cs.ARcited by 5

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Expert Parallelism (EP) permits Mixture of Experts (MoE) models to scale beyond a single GPU. To address load imbalance across GPUs in EP, existing approaches aim to balance the number of tokens each GPU processes. Surprisingly, we find that this objective degrades performance rather than improving it when processing is memory-bound - a common occurrence in MoE serving, especially in the decode phase. Our analysis reveals that balancing the number of tokens processed per GPU increases the number of activated experts, exacerbating memory pressure in the memory-bound regime. We propose Minimum Expert Token ROuting, a novel token-routing algorithm for high-performance expert-parallel MoE serving in the memory-bound regime that balances the number of activated experts per GPU rather than token counts. METRO achieves near-optimal routing quality with minimal computational overhead by jointly optimizing algorithmic efficiency and leveraging the GPU's parallel processing power. To guarantee routing quality, METRO also employs a novel allGather scheme to gather global top-k knowledge, which has minimal overhead compared to conventional allToAll. Our evaluation of METRO against EPLB on both real systems (vLLM over 8 A100 GPUs) and a proprietary simulator (8-16 B200 GPUs) shows that METRO reduces decode latency by 11 - 22%, and total token throughput by 3 - 21% for Qwen3 and DeepSeek-V3 serving, where prefill and decode phases are co-deployed. In addition, by trading latency headroom for throughput, METRO improves decode throughput by up to 4.11x over EPLB at a fixed decode SLO.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 28 of its 52 references that are in the map. First posted on arXiv 10 Dec 2025.

Builds on

28 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference16 Jan
  2. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  3. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  4. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 23

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  12. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
11 more
  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  2. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  4. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  5. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  6. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  7. 2025Barbarians at the Gate: How AI is Upending Systems Researchcited by 48
  8. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  9. 2023InstructCoder: Instruction Tuning Large Language Models for Code Editingcited by 28
  10. 2025SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inferencecited by 6
  11. 2025Can Tensor Cores Benefit Memory-Bound Kernels? (No!)cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMscited by 4
  2. 2026Routing-Free Mixture-of-Expertscited by 2
  3. 2026ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Servingcited by 2
  4. 2026Potential Applications of HBF in LLM Serving Systemscited by 2
  5. 2026TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
  2. 2026TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training
  3. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design
  4. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
  5. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
  6. 2026ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving
  7. 2025MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing
  8. 2026Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs