mlsanityData to 14 Oct 2026

HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmission

arXiv 2510.19470 ↗First posted 22 Oct 2025cs.DC cs.AI cs.LGcited by 2

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) has become a popular architecture for scaling large models. However, the rapidly growing scale outpaces model training on a single DC, driving a shift toward a more flexible, cross-DC training paradigm. Under this, Expert Parallelism (EP) of MoE faces significant scalability issues due to the limited cross-DC bandwidth. Specifically, existing EP optimizations attempt to overlap data communication and computation, which has little benefit in low-bandwidth scenarios due to a much longer data communication time. Therefore, the trends of cross-DC EP scaling is fast becoming a critical roadblock to the continued growth of MoE models. To address this, we propose HybridEP, a modeling-guided framework to optimize EP under constrained bandwidth. Our key idea is to dynamically transform the spatial placement of experts to reduce data communication traffic and frequency, thereby minimizing EP's communication overheads. However, it is non-trivial to find the optimal solution because it complicates the original communication pattern by mixing data and expert communication. We therefore build a stream-based model to determine the optimal transmission ratio. Guided by this, we incorporate two techniques: (1) domain-based partition to construct the mapping between hybrid patterns and specific communication topology at GPU level, and (2) parameter-efficient migration to further refine this topology by reducing expert transmission overhead and enlarging the domain size. Combining all these designs, HybridEP can be considered as a more general EP with better scalability. Experimental results show that HybridEP outperforms existing state-of-the-art MoE training systems by up to 5.6x under constrained bandwidth. We further compare HybridEP and EP on large-scale simulations. HybridEP achieves up to 1.45x speedup with 1k DCs under different bandwidths.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 38 of its 61 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

38 of 61 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Serving Large Language Models on Huawei CloudMatrix38415 Jun
  2. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models18 Jan
  3. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  4. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  5. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  6. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2024Mixtral of Expertscited by 1,823
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
20 more
  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  2. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  4. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  5. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  6. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  7. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  8. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 166
  9. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  10. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  11. 2021Distributed Deep Learning in Open Collaborationscited by 58
  12. 2025A Survey of Large Language Model Agents for Question Answeringcited by 48
  13. 2025INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learningcited by 25
  14. 2024INTELLECT-1 Technical Reportcited by 21
  15. 2024BitMoD: Bit-serial Mixture-of-Datatype LLM Accelerationcited by 18
  16. 2024Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real Systemcited by 15
  17. 2022AI for Next Generation Computing: Emerging Trends and Future Directionscited by 13
  18. 2023eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Modelscited by 4
  19. 2024Enhancing Large-Scale AI Training Efficiency: The C4 Solution for Real-Time Anomaly Detection and Communication Optimizationcited by 4
  20. 2025CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Trainingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Trainingcited by 5
  2. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design
  2. 2026HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training
  3. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training
  4. 2026Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts
  5. 2026FastEEP: Fast Intra-Instance Elastic Scaling for Expert-Parallel MoE Serving
  6. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
  7. 2026Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement
  8. 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing