mlsanityData to 7 Oct 2026

DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale

arXiv 2201.05596 ↗First posted 14 Jan 2022cs.LG cs.AI cs.DCcited by 400

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

As the training of giant dense models hits the boundary on the availability and capability of the hardware resources today, Mixture-of-Experts (MoE) models become one of the most promising model architectures due to their significant training cost reduction compared to a quality-equivalent dense model. Its training cost saving is demonstrated from encoder-decoder models (prior works) to a 5x saving for auto-aggressive language models (this work along with parallel explorations). However, due to the much larger model size and unique architecture, how to provide fast MoE model inference remains challenging and unsolved, limiting its practical usage. To tackle this, we present DeepSpeed-MoE, an end-to-end MoE training and inference solution as part of the DeepSpeed library, including novel MoE architecture designs and model compression techniques that reduce MoE model size by up to 3.7x, and a highly optimized inference system that provides 7.3x better latency and cost compared to existing MoE inference solutions. DeepSpeed-MoE offers an unprecedented scale and efficiency to serve massive MoE models with up to 4.5x faster and 9x cheaper inference compared to quality-equivalent dense models. We hope our innovations and systems help open a promising path to new directions in the large model landscape, a shift from dense to sparse MoE models, where training and deploying higher-quality models with fewer resources becomes more widely possible.

Key facts. Cited by 400 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 36 of its 53 references that are in the map. First posted on arXiv 14 Jan 2022.

Builds on

36 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
23 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  10. 2014How transferable are features in deep neural networks?cited by 1,216
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  13. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  14. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  15. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  16. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  17. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  18. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  19. 2021Block Pruning For Faster Transformerscited by 176
  20. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  21. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  22. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  23. 2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 28; 2023: 45; 2024: 88; 2025: 123; 2026: 116.

400 papers.

Sort:

By area:

  1. 2024Qwen2 Technical Reportcited by 2,076
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  5. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  6. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  7. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  8. 2023Efficient Large Language Models: A Surveycited by 183
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  10. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  11. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  12. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
38 more
  1. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  2. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  3. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  4. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  5. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  6. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  7. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  8. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  9. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  10. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  11. 2024Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Expertscited by 88
  12. 2023Modular Deep Learningcited by 82
  13. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  14. 2023LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Modelscited by 71
  15. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  16. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
  17. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 69
  18. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  19. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 65
  20. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  21. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 58
  22. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  23. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  24. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  25. 2025LongCat-Flash Technical Reportcited by 54
  26. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 53
  27. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  28. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 47
  29. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  30. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  31. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  32. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  33. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Expertscited by 46
  34. 2023Making AI Less "Thirsty": Uncovering and Addressing the Secret Water Footprint of AI Modelscited by 44
  35. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  36. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 41
  37. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  38. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37

50 most cited of 400 All 400 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading
  2. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  3. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  4. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
  5. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  6. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  7. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  8. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts