# Mixture-of-Experts Serving

arXiv: https://arxiv.org/abs/2607.17880 · HTML: https://mlsanity.com/paper/2607.17880/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2026.

## Built on by (most cited first, 2 of 2)

- [TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes](https://mlsanity.com/paper/2608.13057/) (2026)
- [A Deterministic Constant-Competitive Algorithm for Dynamic Mixture-of-Experts Serving](https://mlsanity.com/paper/2608.16947/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) models route each token to only a few expert networks, distributing the serving load across experts whose popularity shifts over time. A serving system must therefore dynamically decide how many GPUs to assign to each expert, trading off service latency against the cost of reconfiguring the assignment. We introduce a formal model of MoE Serving and initiate a principled study of online and offline algorithms for it. Our main result is a polynomial-time O(√log k)-competitive online algorithm, where k is the number of GPUs beyond one per expert. We complement it with a matching Omega(√log k) barrier for the online dual problem underlying our analysis. In the offline setting, we give a constant-factor approximation, show that MoE Serving is NP-hard, and rule out an FPTAS assuming ETH.