mlsanityData to 7 Oct 2026

ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving

arXiv 2606.00735 ↗First posted 30 May 2026cs.DC cs.LGcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

In distributed Mixture-of-Experts (MoE) inference, input-dependent token routing interacts with GPU performance variability to create persistent stragglers under synchronized execution, where the slowest GPU determines layer latency. This performance variability is inherent to modern accelerators: manufacturing variation, power limits, and thermal conditions introduce measurable execution-time differences across nominally identical GPUs. The core challenge is that MoE execution-time imbalance arises from the interaction of workload skew and hardware asymmetry. Token routing produces uneven and layer-varying expert loads, while GPU throughput depends on device-specific operating characteristics and workload intensity. Prior work mitigates routing skew but assumes homogeneous hardware, optimizing token balance rather than execution latency. As a result, even balanced token assignments can leave hardware-induced stragglers unaddressed. Thus, we propose Variability-Informed Binning of Experts (ViBE), a hardware-aware expert placement framework that minimizes execution-time imbalance across GPUs. ViBE combines per-GPU performance modeling with expert activation profiling to assign high-load experts to faster devices and low-load experts to slower ones, reducing layer-level stragglers without modifying model semantics or hardware. Because both workload characteristics and effective GPU throughput can shift across serving conditions, ViBE supports lightweight recalibration under workload/performance drift to refresh its routing and performance estimates when needed. Results show that ViBE consistently reduces execution-time imbalance and improves SLO attainment by 14%, while lowering P90 TTFT by up to 45%. We further show that the impact of hardware variability increases at scale, making variability-aware placement important for efficient, high-utilization LLM serving.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 24 of its 43 references that are in the map. First posted on arXiv 30 May 2026.

Builds on

24 of 43 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts27 Feb
  2. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing10 Feb
  3. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference16 Jan
  4. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  6. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 18

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  12. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
6 more
  1. 2023Reducing Energy Bloat in Large Model Trainingcited by 22
  2. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
  3. 2022Not All GPUs Are Created Equal: Characterizing Variability in Large-Scale, Accelerator-Rich Systemscited by 9
  4. 2025Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspectivecited by 5
  5. 2024Integrated Hardware Architecture and Device Placement Searchcited by 4
  6. 2025Lit Silicon: A Case Where Thermal Imbalance Couples Concurrent Execution in Multiple GPUscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens
  2. 2026GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
  3. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
  4. 2025MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing
  5. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
  6. 2026TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training
  7. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  8. 2025HarMoEny: Efficient Multi-GPU Inference of MoE Models