# Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models

arXiv: https://arxiv.org/abs/2501.11873 · HTML: https://mlsanity.com/paper/2501.11873/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 27 of its 30 references that are in the map.
- First posted on arXiv 21 Jan 2025.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [MoE-LLaVA: Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2401.15947/) (2024)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [MegaBlocks: Efficient Sparse Training with Mixture-of-Experts](https://mlsanity.com/paper/2211.15841/) (2022)
- [Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts](https://mlsanity.com/paper/2408.15664/) (2024)
- [Tutel: Adaptive Mixture-of-Experts at Scale](https://mlsanity.com/paper/2206.03382/) (2022)
- [OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2402.01739/) (2024)
- [MiniMax-01: Scaling Foundation Models with Lightning Attention](https://mlsanity.com/paper/2501.08313/) (2025)
- [Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training](https://mlsanity.com/paper/2405.03133/) (2024)
- [Fewer Truncations Improve Language Modeling](https://mlsanity.com/paper/2404.10830/) (2024)
- [Layerwise Recurrent Router for Mixture-of-Experts](https://mlsanity.com/paper/2408.06793/) (2024)
- [Unlocking Emergent Modularity in Large Language Models](https://mlsanity.com/paper/2310.10908/) (2023)
- [GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory](https://mlsanity.com/paper/2406.12375/) (2024)

## Built on by (most cited first, 30 of 54)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://mlsanity.com/paper/2505.06708/) (2025)
- [On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability](https://mlsanity.com/paper/2608.30320/) (2026)
- [Controlled LLM Training on Spectral Sphere](https://mlsanity.com/paper/2601.08393/) (2026)
- [DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers](https://mlsanity.com/paper/2503.14487/) (2025)
- [A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training](https://mlsanity.com/paper/2601.22966/) (2026)
- [Load Balancing Mixture of Experts with Similarity Preserving Routers](https://mlsanity.com/paper/2506.14038/) (2025)
- [MoRE: 3D Visual Geometry Reconstruction Meets Mixture-of-Experts](https://mlsanity.com/paper/2510.27234/) (2025)
- [The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise](https://mlsanity.com/paper/2604.09780/) (2026)
- [Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs](https://mlsanity.com/paper/2505.04519/) (2025)
- [STEM: Scaling Transformers with Embedding Modules](https://mlsanity.com/paper/2601.10639/) (2026)
- [ProPhy: Progressive Physical Alignment for Dynamic World Simulation](https://mlsanity.com/paper/2512.05564/) (2025)
- [Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss](https://mlsanity.com/paper/2512.23447/) (2025)
- [Dense Backpropagation Improves Training for Sparse Mixture-of-Experts](https://mlsanity.com/paper/2504.12463/) (2025)
- [Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts](https://mlsanity.com/paper/2601.17111/) (2026)
- [Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts](https://mlsanity.com/paper/2604.08541/) (2026)
- [ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization](https://mlsanity.com/paper/2511.10971/) (2025)
- [SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training](https://mlsanity.com/paper/2605.08738/) (2026)
- [Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency](https://mlsanity.com/paper/2603.17102/) (2026)
- [φ-Balancing for Mixture-of-Experts Training](https://mlsanity.com/paper/2605.15403/) (2026)
- [PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetching](https://mlsanity.com/paper/2602.00509/) (2026)
- [Expert Divergence Learning for MoE-based Language Models](https://mlsanity.com/paper/2603.00054/) (2026)
- [EMO: Pretraining Mixture of Experts for Emergent Modularity](https://mlsanity.com/paper/2605.06663/) (2026)
- [Point-MoE: Large-Scale Multi-Dataset Training with Mixture-of-Experts for 3D Semantic Segmentation](https://mlsanity.com/paper/2505.23926/) (2025)
- [SD-MoE: Spectral Decomposition for Effective Expert Specialization](https://mlsanity.com/paper/2602.12556/) (2026)
- [Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing](https://mlsanity.com/paper/2603.11535/) (2026)
- [Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts](https://mlsanity.com/paper/2605.28042/) (2026)
- [DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts](https://mlsanity.com/paper/2606.01062/) (2026)
- [Sigma-MoE-Tiny Technical Report](https://mlsanity.com/paper/2512.16248/) (2025)
- [EMoE: Eigenbasis-Guided Routing for Mixture-of-Experts](https://mlsanity.com/paper/2601.12137/) (2026)

## Abstract (arXiv)

This paper revisits the implementation of Load-balancing Loss (LBL) when training Mixture-of-Experts (MoEs) models. Specifically, LBL for MoEs is defined as NE sumi=1NE fi pi, where NE is the total number of experts, fi represents the frequency of expert i being selected, and pi denotes the average gating score of the expert i. Existing MoE training frameworks usually employ the parallel training strategy so that fi and the LBL are calculated within a micro-batch and then averaged across parallel groups. In essence, a micro-batch for training billion-scale LLMs normally contains very few sequences. So, the micro-batch LBL is almost at the sequence level, and the router is pushed to distribute the token evenly within each sequence. Under this strict constraint, even tokens from a domain-specific sequence (textite.g., code) are uniformly routed to all experts, thereby inhibiting expert specialization. In this work, we propose calculating LBL using a global-batch to loose this constraint. Because a global-batch contains much more diverse sequences than a micro-batch, which will encourage load balance at the corpus level. Specifically, we introduce an extra communication step to synchronize fi across micro-batches and then use it to calculate the LBL. Through experiments on training MoEs-based LLMs (up to 42.8B total parameters and 400B tokens), we surprisingly find that the global-batch LBL strategy yields excellent performance gains in both pre-training perplexity and downstream tasks. Our analysis reveals that the global-batch LBL also greatly improves the domain specialization of MoE experts.