mlsanityData to 14 Oct 2026

FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction

arXiv 2608.14205 ↗First posted 14 Aug 2026cs.AI cs.LGcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Load imbalance poses a major bottleneck to the efficiency of expert parallelism in distributed inference of Mixture-of-Experts (MoE) models. The most heavily loaded rank stalls global execution due to skewed routing distributions, directly increasing latency. While offline expert placement can alleviate persistent imbalance, practical multi-task serving workloads exhibit layer- and batch-dependent routing dynamics, making online load balancing indispensable. Existing approaches rely on routing statistics collected after each MoE router, requiring expert weight load or migration to begin only after routing decisions are available, consequently placing migration overhead on the inference critical path. In this work, we observe that online balancing can instead be largely overlapped with computation before target routing (e.g., attention), if routing distributions can be predicted accurately in advance. Therefore, we propose FreeBalance, a lossless online load-balancing framework that overlaps expert migration with preceding computation stages via residual workload prediction. FreeBalance leverages cross-layer similarities in hidden representations within the residual network to build a lightweight workload predictor. This enables proactive expert migration planning before routing decisions are available, creating substantial overlap between weight transfer and computation-heavy pre-routing stages. Furthermore, a cost model constrains the number of swaps to fully hide the synchronization overhead within the available window. Experiments across models and datasets show that FreeBalance reduces the max-to-mean rank load ratio by 32.8% and end-to-end prefill latency by 13.1%. Specifically, our method hides balancing overhead of an average of 5.1 experts per layer, which would otherwise account for about 8.5% of the critical-path latency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 10 of its 19 references that are in the map. First posted on arXiv 14 Aug 2026.

Builds on

10 of 19 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing2 Jun
  2. 2025Muon is Scalable for LLM Training24 Feb
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 7

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  6. 2022Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Serverscited by 6
  7. 2025EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilizationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
  2. 2026Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts
  3. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
  4. 2026When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models
  5. 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models
  6. 2026Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection
  7. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design
  8. 2025Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts