mlsanityData to 7 Oct 2026

Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference

arXiv 2303.06182 ↗First posted 10 Mar 2023cs.DC cs.AR cs.CLcited by 36

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) models have gained popularity in achieving state-of-the-art performance in a wide range of tasks in computer vision and natural language processing. They effectively expand the model capacity while incurring a minimal increase in computation cost during training. However, deploying such models for inference is difficult due to their large size and complex communication pattern. In this work, we provide a characterization of two MoE workloads, namely Language Modeling (LM) and Machine Translation (MT) and identify their sources of inefficiencies at deployment. We propose three optimization techniques to mitigate sources of inefficiencies, namely (1) Dynamic gating, (2) Expert Buffering, and (3) Expert load balancing. We show that dynamic gating improves maximum throughput by 6.21-11.23× for LM, 5.75-10.98× for MT Encoder and 2.58-5.71× for MT Decoder. It also reduces memory usage by up to 1.36× for LM and up to 1.1× for MT. We further propose Expert Buffering, a new caching mechanism that only keeps hot, active experts in GPU memory while buffering the rest in CPU memory. This reduces static memory allocation by up to 1.47×. We finally propose a load balancing methodology that provides additional scalability to the workload.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 30 of its 35 references that are in the map. First posted on arXiv 10 Mar 2023.

Builds on

30 of 35 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022No Language Left Behind: Scaling Human-Centered Machine Translation11 Jul
  2. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  4. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
  5. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2020Efficient Transformers: A Surveycited by 784
  9. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  10. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  12. 2021Hash Layers For Large Sparse Modelscited by 236
11 more
  1. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  2. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  3. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  4. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  5. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  6. 2021Sparse is Enough in Scaling Transformerscited by 86
  7. 2021Go Wider Instead of Deepercited by 56
  8. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  9. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  10. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37
  11. 2021M6-T: Exploring Sparse Expert Models and Beyondcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 11; 2025: 12; 2026: 8.

36 papers.

Sort:

By area:

  1. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  3. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  4. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  5. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  6. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  7. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  8. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 38
  9. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 36
  10. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 33
  11. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  12. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routingcited by 26
24 more
  1. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
  2. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  3. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14
  4. 2025Utility-Driven Speculative Decoding for Mixture-of-Expertscited by 12
  5. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inferencecited by 12
  6. 2024MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Modelscited by 11
  7. 2025Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Schedulingcited by 9
  8. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruningcited by 8
  9. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  10. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Schedulingcited by 7
  11. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5
  12. 2024Kraken: Inherently Parallel Transformers For Efficient Multi-Device Inferencecited by 4
  13. 2025MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancingcited by 4
  14. 2025HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inferencecited by 3
  15. 2026SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decodingcited by 3
  16. 2025MC#: Mixture Compressor for Mixture-of-Experts Large Modelscited by 1
  17. 2026ReaLB: Real-Time Load Balancing for Multimodal MoE Inferencecited by 1
  18. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  19. 2025Efficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborativecited by 0
  20. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMscited by 0
  21. 2026ExFusion: Efficient Transformer Training via Multi-Experts Fusioncited by 0
  22. 2026An Interpretable Latency Model for Speculative Decoding in LLM Servingcited by 0
  23. 2026NASiC: 3D NAND-based CAM-Selected Multibit CIM Architecture for Efficient On-Device Mixture-of-Experts LLM Inferencecited by 0
  24. 2026Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  2. 2025Mixture of Experts in Large Language Models
  3. 2025Faster MoE LLM Inference for Extremely Large Models
  4. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts
  5. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  6. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  7. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
  8. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production