DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
Mixture-of-experts (MoE) architectures enable trillion-parameter LLMs with sparsely activated experts. Expert parallelism (EP) is a widely adopted MoE training strategy, but it suffers from severe all-to-all communication bottlenecks, which is exaggerated by the limited inter-node network bandwidth as the growing model size requires distributing experts across GPU nodes. Prior work focused on overlapping these all-to-all communications with feed-forward network (FFN) and self-attention computations, which often leaves residual network-bound stalls due to inherent imbalance in attention and FFN layers' computation-communication ratios. We present DisagMoE, a disaggregated MoE training system that jointly optimizes model placement and scheduling for maximal efficiency. DisagMoE separates attention and FFN layers into disjoint GPU groups, introduces a multi-stage pipeline with uni-directional, many-to-many communications, and employs a computation-communication roofline model to balance GPU and network bandwidth allocation among the attention and FFN groups. DisagMoE is implemented on Megatron-LM, and evaluation shows that DisagMoE improves training efficiency across multiple MoE models with up to 1.8x speedup on 16-node 8xH800 clusters.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 14 of its 34 references that are in the map. First posted on arXiv 10 May 2026.
Builds on
14 of 34 references.
Closest ancestors · 5
Same direction, newest first
- 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding25 Jul
- 2025HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs4 Apr
- 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism3 Apr
- 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts27 Feb
- 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
Other references · 9
- 2025Qwen3 Technical Reportcited by 8,195
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2024Mixtral of Expertscited by 1,785
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
- 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
- 2024DeepSeek-V3 Technical Reportcited by 724
- 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
- 2026ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels
- 2025Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
- 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
- 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System
- 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
- 2026Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference
- 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts