# RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Training

arXiv: https://arxiv.org/abs/2510.19262 · HTML: https://mlsanity.com/paper/2510.19262/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 18 of its 76 references that are in the map.
- First posted on arXiv 22 Oct 2025.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [An Extensible Software Transport Layer for GPU Networking](https://mlsanity.com/paper/2504.17307/) (2025)
- [Revisiting Network Support for RDMA](https://mlsanity.com/paper/1806.08159/) (2018)
- [Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling](https://mlsanity.com/paper/2410.17043/) (2024)
- [MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Training](https://mlsanity.com/paper/2501.03905/) (2025)
- [REPS: Recycled Entropy Packet Spraying for Adaptive Load Balancing and Failure Mitigation](https://mlsanity.com/paper/2407.21625/) (2024)
- [STrack: A Reliable Multipath Transport for AI/ML Clusters](https://mlsanity.com/paper/2407.15266/) (2024)
- [Unlocking Diversity of Fast-Switched Optical Data Center Networks with Unified Routing](https://mlsanity.com/paper/2412.00266/) (2024)

## Built on by (most cited first, 1 of 1)

- [φ-Balancing for Mixture-of-Experts Training](https://mlsanity.com/paper/2605.15403/) (2026)

## Abstract (arXiv)

Training Mixture-of-Experts (MoE) models introduces sparse and highly imbalanced all-to-all communication that dominates iteration time. Conventional load-balancing methods fail to exploit the deterministic topology of Rail architectures, leaving multi-NIC bandwidth underutilized. We present RailS, a distributed load-balancing framework that minimizes all-to-all completion time in MoE training. RailS leverages the Rail topology's symmetry to prove that uniform sending ensures uniform receiving, transforming global coordination into local scheduling. Each node independently executes a Longest Processing Time First (LPT) spraying scheduler to proactively balance traffic using local information. RailS activates N parallel rails for fine-grained, topology-aware multipath transmission. Across synthetic and real-world MoE workloads, RailS improves bus bandwidth by 20%--78% and reduces completion time by 17%--78%. For Mixtral workloads, it shortens iteration time by 18%--40% and achieves near-optimal load balance, fully exploiting architectural parallelism in distributed training.