# Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts

arXiv: https://arxiv.org/abs/2601.17111 · HTML: https://mlsanity.com/paper/2601.17111/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 23 Jan 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 12 of 12)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models](https://mlsanity.com/paper/2501.11873/) (2025)
- [Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision](https://mlsanity.com/paper/2512.15489/) (2025)
- [Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler](https://mlsanity.com/paper/2504.19442/) (2025)
- [Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization](https://mlsanity.com/paper/2506.13541/) (2025)

## Built on by (most cited first, 9 of 9)

- [ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving](https://mlsanity.com/paper/2607.00466/) (2026)
- [Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models](https://mlsanity.com/paper/2607.01844/) (2026)
- [TAOT: Topology-Aware Optimal Transport for Dynamic Expert Replica Placement in MoE Training](https://mlsanity.com/paper/2608.03676/) (2026)
- [Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory](https://mlsanity.com/paper/2605.20982/) (2026)
- [TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes](https://mlsanity.com/paper/2608.13057/) (2026)
- [Performance Foundations of Parallel & Distributed Reasoning Language Models](https://mlsanity.com/paper/2608.27046/) (2026)
- [Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training](https://mlsanity.com/paper/2609.14306/) (2026)
- [TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training](https://mlsanity.com/paper/2609.35481/) (2026)
- [CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training](https://mlsanity.com/paper/2610.05744/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) models are typically pre-trained with explicit load-balancing constraints to ensure statistically balanced expert routing. Despite this, we observe that even well-trained MoE models exhibit significantly imbalanced routing. This behavior is arguably natural-and even desirable - as imbalanced routing allows models to concentrate domain-specific knowledge within a subset of experts. Expert parallelism (EP) is designed to scale MoE models by distributing experts across multiple devices, but with a less-discussed assumption of balanced routing. Under extreme imbalance, EP can funnel a disproportionate number of tokens to a small number of experts, leading to compute- and memory-bound failures on overloaded devices during post-training or inference, where explicit load balancing is often inapplicable. We propose Least-Loaded Expert Parallelism (LLEP), a novel EP algorithm that dynamically reroutes excess tokens and associated expert parameters from overloaded devices to underutilized ones. This ensures that all devices complete their workloads within the minimum collective latency while respecting memory constraints. Across different model scales, LLEP achieves up to 5x speedup and 4x reduction in peak memory usage compared to standard EP. This enables faster and higher-throughput post-training and inference, with ~1.9x faster for gpt-oss-120b. We support our method with extensive theoretical analysis and comprehensive empirical evaluations, including ablation studies. These results illuminate key trade-offs and enable a principled framework for hardware-specific hyper-parameter tuning to achieve optimal performance.