mlsanityData to 14 Oct 2026

A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training

arXiv 2303.06318 ↗First posted 11 Mar 2023cs.LG cs.AI cs.DCcited by 47

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) is a neural network architecture that adds sparsely activated expert blocks to a base model, increasing the number of parameters without impacting computational costs. However, current distributed deep learning frameworks are limited in their ability to train high-quality MoE models with large base models. In this work, we present DeepSpeed-TED, a novel, three-dimensional, hybrid parallel algorithm that combines data, tensor, and expert parallelism to enable the training of MoE models with 4 to 8x larger base models than the current state-of-the-art. We also describe memory optimizations in the optimizer step, and communication optimizations that eliminate unnecessary data movement. We implement our approach in DeepSpeed and achieve speedups of 26% over a baseline (i.e. without our communication optimizations) when training a 40 billion parameter MoE model (6.7 billion base model with 16 experts) on 128 V100 GPUs.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 34 of its 45 references that are in the map. First posted on arXiv 11 Mar 2023.

Builds on

34 of 45 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  2. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May
  3. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System28 Mar
  4. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2017Decoupled Weight Decay Regularizationcited by 1,831
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
18 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  2. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  3. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  4. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  5. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  6. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  7. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  8. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  9. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 100
  10. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 90
  11. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  12. 2021Go Wider Instead of Deepercited by 57
  13. 2020Reducing Communication in Graph Neural Network Trainingcited by 50
  14. 2021Maximizing Parallelism in Distributed Training for Huge Neural Networkscited by 23
  15. 2021Tesseract: Parallelize the Tensor Parallelism Efficientlycited by 19
  16. 2021AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learningcited by 15
  17. 2019Improving Strong-Scaling of CNN Training by Exploiting Finer-Grained Parallelismcited by 8
  18. 2021Automatic Graph Partitioning for Very Large-scale Deep Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 26; 2026: 7.

47 papers.

Sort:

By area:

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  2. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 32
  3. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 32
  4. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inferencecited by 28
  5. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routingcited by 26
  6. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  7. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Corecited by 17
  8. 2024SmileyLlama: Modifying Large Language Models for Directed Chemical Space Explorationcited by 14
  9. 2025FlashMoE: Fast Distributed MoE in a Single Kernelcited by 14
  10. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  11. 2025PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Servingcited by 9
  12. 2025Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Schedulingcited by 9
35 more
  1. 2024Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learningcited by 8
  2. 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedulescited by 7
  3. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  4. 2025Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputerscited by 7
  5. 2025HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUscited by 7
  6. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inferencecited by 6
  7. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Modelscited by 6
  8. 2024Theory of Mixture-of-Experts for Mobile Edge Computingcited by 5
  9. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decouplingcited by 5
  10. 2025Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspectivecited by 5
  11. 2023TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learningcited by 4
  12. 2024MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Trainingcited by 4
  13. 2026Frontier: Towards Comprehensive and Accurate LLM Inference Simulationcited by 4
  14. 2025X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platformscited by 3
  15. 2025HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inferencecited by 3
  16. 2025Frontier: Simulating the Next Generation of LLM Inference Systemscited by 2
  17. 2025BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Modelscited by 2
  18. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  19. 2025iServe: An Intent-based Serving System for LLMscited by 1
  20. 2025Maya: Optimizing Deep Learning Training Workloads using GPU Runtime Emulationcited by 1
  21. 2025DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstructioncited by 1
  22. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 1
  23. 2025TrueMoE: Dual-Routing Mixture of Discriminative Experts for Synthetic Image Detectioncited by 1
  24. 2025Accelerating Frontier MoE Training with 3D Integrated Opticscited by 1
  25. 2025Design Space Exploration of DMA based Finer-Grain Compute Communication Overlapcited by 1
  26. 2026HetCCL: Accelerating LLM Training with Heterogeneous GPUscited by 1
  27. 2026Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switchcited by 1
  28. 2026On the Design of Mixture-of-Experts for Dynamic Gaussian Splattingcited by 1
  29. 2024MoNTA: Accelerating Mixture-of-Experts Training with Network-Traffc-Aware Parallel Optimizationcited by 0
  30. 2025Accelerating MoE Model Inference with Expert Shardingcited by 0
  31. 2025The Big Send-off: Scalable and Performant Collectives for Deep Learningcited by 0
  32. 2025Advancing Compositional LLM Reasoning with Structured Task Relations in Interactive Multimodal Communicationscited by 0
  33. 2025TASP: Topology-aware Sequence Parallelismcited by 0
  34. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0
  35. 2026Federation of Experts: Communication Efficient Distributed Inference for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  2. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core
  3. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  4. 2026MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
  5. 2021FastMoE: A Fast Mixture-of-Expert Training System
  6. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  7. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
  8. 2022Accelerating Distributed MoE Training and Inference with Lina