Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
AlltoAll dispatch is the dominant bottleneck of MoE expert parallelism, and the interconnect community has responded with four families of mitigations: predictive sample placement, adaptive expert relayout, hierarchical collectives, and EP-aware topology. All four rest on two assumptions about the workload: that routing imbalance is correctable by the system layer, and that the mock-token benchmarks evaluating them faithfully represent production routing. We introduce DODOCO to test both, instrumenting five open MoE checkpoints that span today's sequence-mixer designs (MHA, MLA, GQA, Gated DeltaNet and Mamba-2 SSM) under a factorial grid of six data conditions and a matched expert-parallelism scan on H100 clusters. Both assumptions fail. Scaling EP leaves per-expert load concentration essentially unchanged: the straggler is intrinsic to the routing decision the model makes, not to how its experts land on ranks. Mock tokens overestimate routing imbalance by up to a factor of 2.35, and the error is a level offset rather than a trend: it stays flat across a 32× batch-size sweep. Skewing the synthetic distribution toward realism (Zipf) widens the gap instead of closing it. A third pattern organizes the results: the architectures separate into a data-resilient band (MHA, Mamba-2), whose routing approaches uniform on real text, and a persistently concentrated band (MLA, GDN), with GQA intermediate. These bands, not the EP degree or the mock-data profile, are the right workload input to AlltoAll-aware interconnect and dispatch design.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 36 of its 45 references that are in the map. First posted on arXiv 20 May 2026.
Builds on
36 of 45 references.
Closest ancestors · 11
Same direction, newest first
- 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training12 Feb
- 2026Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts23 Jan
- 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference7 Oct
- 2025From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing29 Sep
- 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models24 Aug
- 2025HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs4 Apr
- 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design2 Apr
- 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing10 Feb
- 2023Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters22 Jul
- 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
- 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
Other references · 25
- 2025Qwen3 Technical Reportcited by 8,820
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
- 2022Mixture-of-Experts with Expert Choice Routingcited by 623
- 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
- 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
- 2021Hash Layers For Large Sparse Modelscited by 237
- 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
13 more
- 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
- 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 53
- 2025Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundationcited by 34
- 2025Multilingual Routing in Mixture-of-Expertscited by 29
- 2024Towards an empirical understanding of MoE design choicescited by 20
- 2024Demystifying the Communication Characteristics for Distributed Transformer Modelscited by 7
- 2025Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Expertscited by 4
- 2026Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Trainingcited by 4
- 2024Towards a Standardized Representation for Deep Learning Collective Algorithmscited by 3
- 2025Characterizing Communication Patterns in Distributed Large Language Model Inferencecited by 3
- 2025On the Spatial Structure of Mixture-of-Experts in Transformerscited by 2
- 2026Geometric Routing Enables Causal Expert Control in Mixture of Expertscited by 2
- 2025Accelerating Frontier MoE Training with 3D Integrated Opticscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes
- 2026RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts
- 2026TerraceMoE: A Cost Model for Hierarchical MoE All-to-All Communication
- 2026Memory-Efficient Expert Routing for Distributed MoE Training
- 2026EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
- 2026Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale
- 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing
- 2026FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction