mlsanityData to 7 Oct 2026

Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts

arXiv 2408.15664 ↗First posted 28 Aug 2024cs.LG cs.CLcited by 191

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

For Mixture-of-Experts (MoE) models, an unbalanced expert load will lead to routing collapse or increased computational overhead. Existing methods commonly employ an auxiliary loss to encourage load balance, but a large auxiliary loss will introduce non-negligible interference gradients into training and thus impair the model performance. In order to control load balance while not producing undesired gradients during training, we propose Loss-Free Balancing, featured by an auxiliary-loss-free load balancing strategy. To be specific, before the top-K routing decision, Loss-Free Balancing will first apply an expert-wise bias to the routing scores of each expert. By dynamically updating the bias of each expert according to its recent load, Loss-Free Balancing can consistently maintain a balanced distribution of expert load. In addition, since Loss-Free Balancing does not produce any interference gradients, it also elevates the upper bound of model performance gained from MoE training. We validate the performance of Loss-Free Balancing on MoE models with up to 3B parameters trained on up to 200B tokens. Experimental results show that Loss-Free Balancing achieves both better performance and better load balance compared with traditional auxiliary-loss-controlled load balancing strategies.

Key facts. Cited by 191 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 10 of its 10 references that are in the map. First posted on arXiv 28 Aug 2024.

Builds on

10 of 10 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 8

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  8. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 81; 2026: 107.

191 papers.

Sort:

By area:

  1. 2024DeepSeek-V3 Technical Reportcited by 724
  2. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  3. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  4. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  5. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  6. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  7. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  8. 2025mHC: Manifold-Constrained Hyper-Connectionscited by 76
  9. 2025LongCat-Flash Technical Reportcited by 54
  10. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 53
  11. 2026The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligencecited by 50
  12. 2025Advancing Expert Specialization for Better MoEcited by 37
38 more
  1. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 33
  2. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  3. 2026Controlled LLM Training on Spectral Spherecited by 28
  4. 2025Load Balancing Mixture of Experts with Similarity Preserving Routerscited by 22
  5. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 20
  6. 2025RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoningcited by 19
  7. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 16
  8. 2025Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Expertscited by 13
  9. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  10. 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligencecited by 13
  11. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 12
  12. 2025Scaling Linear Attention with Sparse State Expansioncited by 12
  13. 2025Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMscited by 12
  14. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Expertscited by 12
  15. 2025Towards Fully FP8 GEMM LLM Training at Scalecited by 11
  16. 2025Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Losscited by 11
  17. 2026Rethinking Language Model Scaling under Transferable Hypersphere Optimizationcited by 11
  18. 2025Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectivescited by 10
  19. 2025MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptationcited by 10
  20. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Expertscited by 10
  21. 2026STEM: Scaling Transformers with Embedding Modulescited by 10
  22. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 10
  23. 2026Arcee Trinity Large Technical Reportcited by 10
  24. 2025MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Seriescited by 9
  25. 2025Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoningcited by 9
  26. 2025MoE-DP: An MoE-Enhanced Diffusion Policy for Robust Long-Horizon Robotic Manipulation with Skill Decomposition and Failure Recoverycited by 9
  27. 2026ERNIE 5.0 Technical Reportcited by 9
  28. 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertisecited by 9
  29. 2025Dense Backpropagation Improves Training for Sparse Mixture-of-Expertscited by 8
  30. 2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Modelscited by 8
  31. 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancingcited by 8
  32. 2025Autonomy-of-Experts Modelscited by 7
  33. 2025DivControl: Knowledge Diversion for Controllable Image Generationcited by 7
  34. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Expertscited by 7
  35. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  36. 2025Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Expertscited by 6
  37. 2025BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsitycited by 6
  38. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanismscited by 6

50 most cited of 191 All 191 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction
  2. 2026EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
  3. 2026Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts
  4. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training
  5. 2026Distributionally Robust Mixture-of-Experts Training
  6. 2025Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts
  7. 2026When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models
  8. 2026Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection