mlsanityData to 7 Oct 2026

Accelerating MoE Model Inference with Expert Sharding

arXiv 2503.08467 ↗First posted 11 Mar 2025cs.LG cs.AI cs.DCcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture of experts (MoE) models achieve state-of-the-art results in language modeling but suffer from inefficient hardware utilization due to imbalanced token routing and communication overhead. While prior work has focused on optimizing MoE training and decoder architectures, inference for encoder-based MoE models in a multi-GPU with expert parallelism setting remains underexplored. We introduce MoEShard, an inference system that achieves perfect load balancing through tensor sharding of MoE experts. Unlike existing approaches that rely on heuristic capacity factors or drop tokens, MoEShard evenly distributes computation across GPUs and ensures full token retention, maximizing utilization regardless of routing skewness. We achieve this through a strategic row- and column-wise decomposition of expert matrices. This reduces idle time and avoids bottlenecks caused by imbalanced expert assignments. Furthermore, MoEShard minimizes kernel launches by fusing decomposed expert computations, significantly improving throughput. We evaluate MoEShard against DeepSpeed on encoder-based architectures, demonstrating speedups of up to 6.4× in time to first token (TTFT). Our results show that tensor sharding, when properly applied to experts, is a viable and effective strategy for efficient MoE inference.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 51 of its 91 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

51 of 91 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference16 Jan
  2. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training11 Mar
  3. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  4. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  6. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2019Towards Federated Learning at Scale: System Designcited by 826
  11. 2018LEAF: A Benchmark for Federated Settingscited by 780
  12. 2020The Future of Digital Health with Federated Learningcited by 625
33 more
  1. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  2. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  3. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  4. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  5. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  6. 2021Federated Learning for Smart Healthcare: A Surveycited by 182
  7. 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 137
  8. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  9. 2019Reliable Federated Learning for Mobile Networkscited by 92
  10. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 89
  11. 2023Decentralized Federated Learning: A Survey and Perspectivecited by 86
  12. 2019SAFA: a Semi-Asynchronous Protocol for Fast Federated Learning with Low Overheadcited by 82
  13. 2019AI Benchmark: All About Deep Learning on Smartphones in 2019cited by 76
  14. 2021Exponential Graph is Provably Efficient for Decentralized Deep Trainingcited by 55
  15. 2019Federated Learning Of Out-Of-Vocabulary Wordscited by 51
  16. 2021Consensus Control for Decentralized Deep Learningcited by 49
  17. 2019Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirectcited by 41
  18. 2019Personalization of End-to-end Speech Recognition On Mobile Devices For Named Entitiescited by 41
  19. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  20. 2019Federated Learning of N-gram Language Modelscited by 33
  21. 2022Beyond spectral gap: The role of the topology in decentralized learningcited by 27
  22. 2021Resource-Efficient Federated Learningcited by 26
  23. 2022Communication-Efficient Topologies for Decentralized Learning with O(1) Consensus Ratecited by 25
  24. 2023Epidemic Learning: Boosting Decentralized Learning with Randomized Communicationcited by 21
  25. 2023Decentralized Learning Made Easy with DecentralizePycited by 18
  26. 2021D-Cliques: Compensating for Data Heterogeneity with Topology in Decentralized Federated Learningcited by 12
  27. 2024Noiseless Privacy-Preserving Decentralized Learningcited by 9
  28. 2023Beyond Exponential Graph: Communication-Efficient Topologies for Decentralized Learning via Finite-time Convergencecited by 8
  29. 2023A2CiD2: Accelerating Asynchronous Communication in Decentralized Deep Learningcited by 7
  30. 2024Boosting Asynchronous Decentralized Learning with Model Fragmentationcited by 6
  31. 2024PeerSwap: A Peer-Sampler with Randomness Guaranteescited by 3
  32. 2025Scalable Decentralized Learning with Teleportationcited by 3
  33. 2021scriptstyleBASALT: A Rock-Solid Foundation for Epidemic Consensus Algorithms in Very Large, Very Open Networkscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  2. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  3. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
  4. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
  5. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
  6. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  7. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  8. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference