mlsanityData to 14 Oct 2026

MoNTA: Accelerating Mixture-of-Experts Training with Network-Traffc-Aware Parallel Optimization

arXiv 2411.00662 ↗First posted 1 Nov 2024cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The Mixture of Experts (MoE) is an advanced model architecture in the industry that combines multiple specialized expert models from various domains into a single supermodel. This approach enables the model to scale without significantly increasing the computational costs of training and inference, while maximizing model performance. However, current distributed training frameworks do not consider the ultimate optimization of communication, especially for large base models. This paper proposes a network-traffic-aware parallel optimization method that selects the optimal parallel strategy based on the communication volume, and the training cluster's inter-node and intra-node network topologies. Compared to the DeepSpeed, MoNTA achieves an 8x increase in AllToAll communication performance under 8-card tensor parallelism. Compared to the baseline, training a 2x70B model using 16 A800 cards, with an 8K sequence, results in a 13% overall latency performance improvement. Project Page: https://github.com/EnflameTechnology/DeepSpeed.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 11 of its 17 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

11 of 17 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training11 Mar
  2. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  3. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  4. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System28 Mar

Other references · 7

  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  5. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  6. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  7. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  2. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
  3. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  4. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training
  5. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core
  6. 2022SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routing
  7. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference
  8. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling