# Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts

arXiv: https://arxiv.org/abs/2506.21328 · HTML: https://mlsanity.com/paper/2506.21328/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 3 of its 20 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 3 of 3)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)

## Built on by (most cited first, 4 of 4)

- [φ-Balancing for Mixture-of-Experts Training](https://mlsanity.com/paper/2605.15403/) (2026)
- [Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models](https://mlsanity.com/paper/2511.10676/) (2025)
- [Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2605.12476/) (2026)
- [Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory](https://mlsanity.com/paper/2605.20982/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) architectures have emerged as a key strategy for scaling large language models (LLMs) efficiently. However, current MoE systems suffer from severe load imbalance, where only a small subset of experts is consistently activated during training and inference, leading to significant underutilization of model capacity and computational resources. In this work, we revisit expert routing through a clustering perspective and propose Latent Prototype Routing (LPR), a novel routing framework that generalizes existing approaches while promoting balanced expert utilization without compromising downstream performance. Extensive experiments across multiple open-source MoE models -- including DeepSeek-V3, Qwen3-MoE, and Mixtral -- demonstrate that LPR reduces the Gini coefficient of expert load from 0.70 to 0.035 on average, improves the min-max expert load ratio from 1e-6 to 0.70, achieving near-perfect load balancing.