mlsanityData to 7 Oct 2026

Load Balancing Mixture of Experts with Similarity Preserving Routers

arXiv 2506.14038 ↗First posted 16 Jun 2025cs.LGcited by 22

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparse Mixture of Experts (MoE) models offer a scalable and efficient architecture for training large neural networks by activating only a subset of parameters ("experts") for each input. A learned router computes a distribution over these experts, and assigns input tokens to a small subset. However, without auxiliary balancing mechanisms, routers often converge to using only a few experts, severely limiting model capacity and degrading performance. Most current load balancing mechanisms encourage a distribution over experts that resembles a roughly uniform distribution of experts per token. During training, this can result in inconsistent routing behavior, resulting in the model spending its capacity to learn redundant knowledge. We address this by introducing a novel load balancing loss that preserves token-wise relational structure, encouraging consistent expert choices for similar inputs during training. Our experimental results show that applying our loss to the router results in 36% faster convergence and lower redundancy compared to a popular load balancing loss.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 30 of its 38 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

30 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts28 Aug
  2. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2020GLU Variants Improve Transformercited by 1,724
15 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2019Root Mean Square Layer Normalizationcited by 1,425
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  6. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  7. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  9. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  10. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  11. 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 78
  12. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 53
  13. 2023Multi-Task Reinforcement Learning with Mixture of Orthogonal Expertscited by 45
  14. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 28
  15. 2023Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversioncited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 19.

22 papers.

Sort:

By area:

  1. 2025Mixture of Experts in Large Language Modelscited by 15
  2. 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertisecited by 9
  3. 2026KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Modelscited by 7
  4. 2025From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routingcited by 5
  5. 2026φ-Balancing for Mixture-of-Experts Trainingcited by 4
  6. 2026Routing-Free Mixture-of-Expertscited by 2
  7. 2026Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers:Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlockcited by 2
  8. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scalecited by 1
  9. 2026SemLT3D: Semantic-Guided Expert Distillation for Camera-only Long-Tailed 3D Object Detectioncited by 1
  10. 2026LiMoDE: Rethinking Lifelong Robot Manipulation from a Mixture-of-Dynamic-Experts Perspectivecited by 1
  11. 2026Multi-level context Modeling for consistent expert selection in Mixture-of-Expertscited by 1
  12. 2025Mixture of Raytraced Expertscited by 0
10 more
  1. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMscited by 0
  2. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  3. 2026Multi-Domain Learning with Global Expert Mappingcited by 0
  4. 2026Eradicating Negative Transfer in Multi-Physics Foundation Models via Sparse Mixture-of-Experts Routingcited by 0
  5. 2026Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thoughtcited by 0
  6. 2026Hierarchical Copula-Gumbel-Top-K Routing: Two-Sided Dependence Control for Frozen Mixture-of-Experts at Fixed Per-Token Routing Lawscited by 0
  7. 2026Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillationcited by 0
  8. 2026In Two Minds about Lifelong Learning: Exploring Hemispheric Redundancy and Specialisation in Neural Modelscited by 0
  9. 2026When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Modelscited by 0
  10. 2026HERO-MoE: Historical Expert Routing with Scale-Preserving Fusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimization
  2. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routing
  3. 2024Routers in Vision Mixture of Experts: An Empirical Study
  4. 2025A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models
  5. 2026MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
  6. 2026Towards a Statistical Understanding of Mixture-of-Experts
  7. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  8. 2026Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts