mlsanityData to 7 Oct 2026

MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systems

arXiv 2605.05888 ↗First posted 7 May 2026cs.AR cs.DCcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The Mixture-of-Experts (MoE) architecture is crucial for scaling large language models, but its scalability is severely limited by inter-GPU communication bottlenecks in multi-GPU systems. Although overlapping communication with computation is a widely recognized optimization, its effective deployment still remains challenging, both in terms of performance and programmability. In this work, we identify the root cause as a fundamental abstraction mismatch between MoE's dynamic, irregular token-to-expert mapping and the static, address-centric communication model of modern GPUs, which necessitates a complex software mediation phase to resolve addresses before data transfers, limiting performance and software flexibility. To resolve this, we propose MoE-Hub, a hardware-software co-design that introduces a destination-agnostic communication paradigm. MoE-Hub decouples data transmission from address management, allowing producers to send data immediately after routing using only a logical destination, while address allocation and data-flow orchestration are handled transparently by lightweight hardware in the GPU hub. By hardware-accelerating the entire communication control plane, MoE-Hub enables seamless and transparent overlap. Our evaluation shows that MoE-Hub achieves 1.40x-3.08x per-layer and 1.21x-1.98x end-to-end speedup over state-of-the-art systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 32 of its 63 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

32 of 63 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025FlashMoE: Fast Distributed MoE in a Single Kernel5 Jun
  2. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production16 May
  3. 2025Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs7 May
  4. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts27 Feb
  5. 2024Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping30 Apr
  6. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  7. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  8. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  9. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  10. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May
  11. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 21

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2024Mixtral of Expertscited by 1,785
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  9. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  11. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  12. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
9 more
  1. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  2. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  3. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  4. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  5. 2021Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloadscited by 40
  6. 2024T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectivescited by 15
  7. 2024GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitchingcited by 13
  8. 2025Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reorderingcited by 9
  9. 2025Practical Federated Learning without a Servercited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs
  2. 2026Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs
  3. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
  4. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
  5. 2026Democratizing MoE inference on commodity GPUs with CoMoE
  6. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens
  7. 2026Weave: Fine-Grained Dynamic SM Scheduling in an MoE Megakernel for Compute-Communication Overlap
  8. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware