mlsanityData to 14 Oct 2026

φ-Balancing for Mixture-of-Experts Training

arXiv 2605.15403 ↗First posted 14 May 2026cs.LG math.OC stat.MLcited by 5

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) models rely on balanced expert utilization to fully realize their scalability. However, existing load-balancing methods are largely heuristic and operate on noisy mini-batch assignment statistics, introducing bias relative to population-level objectives. We propose φ-balancing, a principled framework that directly targets population-level expert balance by minimizing a strictly convex, symmetric, and differentiable potential of the expected routing distribution. Using convex duality, we derive an equivalent min-max formulation and obtain a simple online algorithm via mirror descent, yielding an efficient EMA-based routing adjustment with negligible overhead. Across large-scale pretraining and downstream fine-tuning, φ-balancing consistently outperforms prior Switch-style and loss-free baselines, demonstrating more stable and effective expert utilization.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 56 of its 69 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

56 of 69 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss29 Dec
  2. 2025ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization14 Nov
  3. 2025Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts26 Jun
  4. 2025Load Balancing Mixture of Experts with Similarity Preserving Routers16 Jun
  5. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing19 Dec
  6. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts28 Aug
  7. 2024Layerwise Recurrent Router for Mixture-of-Experts13 Aug
  8. 2024AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models19 Jun
  9. 2022StableMoE: Stable Routing Strategy for Mixture of Experts18 Apr
  10. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan
  12. 2019Hierarchical Routing Mixture of Experts18 Mar

Other references · 44

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  11. 2020GLU Variants Improve Transformercited by 1,804
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
32 more
  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  3. 2018Neural Network Acceptability Judgmentscited by 958
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  5. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  6. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  7. 2025Muon is Scalable for LLM Trainingcited by 428
  8. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  9. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  10. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  11. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  12. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  13. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  14. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 54
  15. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  16. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 47
  17. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 45
  18. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
  19. 2025Advancing Expert Specialization for Better MoEcited by 39
  20. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 29
  21. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 24
  22. 2024Communication Efficient Distributed Training with Distributed Lioncited by 18
  23. 2025Mixture of Experts in Large Language Modelscited by 15
  24. 2025Cautious Weight Decaycited by 13
  25. 2024Memory-Efficient Optimization with Factorized Hamiltonian Descentcited by 11
  26. 2024DeMo: Decoupled Momentum Optimizationcited by 11
  27. 2026Learning Robust Reasoning through Guided Adversarial Self-Playcited by 8
  28. 2025Revisit Visual Prompt Tuning: The Expressiveness of Prompt Expertscited by 5
  29. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 4
  30. 2025FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computingcited by 2
  31. 2025RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Trainingcited by 1
  32. 2025SAGE: Shape-Adapting Gated Experts for Adaptive Histopathology Image Segmentationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Training-Free Looped Transformerscited by 15
  2. 2026EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMscited by 1
  3. 2026How to Loop MoE: Flatten the Experts, Untie the Attentioncited by 1
  4. 2026Confidence-Adaptive SwiGLU for Mixture-of-Expertscited by 0
  5. 2026CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models
  2. 2025Binary-Integer-Programming Based Algorithm for Expert Load Balancing in Mixture-of-Experts Models
  3. 2026Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts
  4. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
  5. 2026Routing-Free Mixture-of-Experts
  6. 2026When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models
  7. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimization
  8. 2026FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction