mlsanityData to 7 Oct 2026

Exact Quantile Balancing and Load-Error Injection for Mixture-of-Experts

arXiv 2609.28053 ↗First posted 23 Sep 2026cs.LG cs.CLcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems also fits: Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) training requires global load balance to prevent expert under-utilization and local balance for efficient expert-parallel execution. Existing distributed Quantile Balancing (QB) uses shard-dependent or approximate global quantiles, while token-independent expert biases cannot ensure microbatch-level balance. We introduce Exact Quantile Balancing (EQB), which computes exact global-batch BF16 quantiles with negligible communication, and Load-Error Injection (LEI), which injects local load errors directly into router-score gradients. On 7.5B-parameter MoEs trained for up to 500B tokens, EQB improves global balance and downstream performance over naive QB, while LEI improves local balance and outperforms the GShard loss at comparable quality.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 12 of its 17 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

12 of 17 references.

References · 12

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
  12. 2025RadiK: Scalable and Optimized GPU-Parallel Radix Top-K Selectioncited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training
  2. 2026Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts
  3. 2026EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
  4. 2026φ-Balancing for Mixture-of-Experts Training
  5. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
  6. 2026FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction
  7. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
  8. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling