mlsanityData to 7 Oct 2026

A Survey on Inference Optimization Techniques for Mixture of Experts Models

arXiv 2412.14219 ↗First posted 18 Dec 2024cs.LG cs.AI cs.DCcited by 46

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The emergence of large-scale Mixture of Experts (MoE) models represents a significant advancement in artificial intelligence, offering enhanced model capacity and computational efficiency through conditional computation. However, deploying and running inference on these models presents significant challenges in computational resources, latency, and energy efficiency. This comprehensive survey analyzes optimization techniques for MoE models across the entire system stack. We first establish a taxonomical framework that categorizes optimization approaches into model-level, system-level, and hardware-level optimizations. At the model level, we examine architectural innovations including efficient expert design, attention mechanisms, various compression techniques such as pruning, quantization, and knowledge distillation, as well as algorithm improvement including dynamic routing strategies and expert merging methods. At the system level, we investigate distributed computing approaches, load balancing mechanisms, and efficient scheduling algorithms that enable scalable deployment. Furthermore, we delve into hardware-specific optimizations and co-design strategies that maximize throughput and energy efficiency. This survey provides both a structured overview of existing solutions and identifies key challenges and promising research directions in MoE inference optimization. To facilitate ongoing updates and the sharing of cutting-edge advances in MoE inference optimization research, we have established a repository accessible at https://github.com/MoE-Inf/awesome-moe-inference/.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 159 of its 250 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

159 of 250 references.

Closest ancestors · 31

Same direction, newest first

  1. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference27 Nov
  2. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  3. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection13 Nov
  4. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  5. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  6. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design24 Oct
  7. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling22 Oct
  8. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference16 Oct
  9. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts9 Oct
  10. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batching2 Sep
  11. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
  12. 2024MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training8 Aug
19 more
  1. 2024Mixture of Experts with Mixture of Precisions for Tuning Quality of Service19 Jul
  2. 2024Lazarus: Resilient and Elastic Training of Mixture-of-Experts Models5 Jul
  3. 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules30 Jun
  4. 2024MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Models29 May
  5. 2024Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing25 Apr
  6. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts7 Apr
  7. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  8. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference16 Jan
  9. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  10. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  11. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices28 Aug
  12. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  13. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training11 Mar
  14. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  15. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  16. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May
  17. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System28 Mar
  18. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  19. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 128

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2024Mixtral of Expertscited by 1,785
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  13. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  14. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  15. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  16. 2023LIMA: Less Is More for Alignmentcited by 1,080
  17. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  18. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  19. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  20. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  21. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  22. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  23. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  24. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  25. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  26. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  27. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  28. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  29. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  30. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  31. 2023From Sparse to Soft Mixtures of Expertscited by 245
  32. 2021Hash Layers For Large Sparse Modelscited by 236
  33. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  34. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  35. 2019MLPerf Inference Benchmarkcited by 195
  36. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  37. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  38. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185

50 most cited of 128 All 128 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 27.

46 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 40
  3. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrationscited by 16
  4. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  5. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Expertscited by 11
  6. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  7. 2025Safety Evaluation of DeepSeek Models in Chinese Contextscited by 9
  8. 2025EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimizationcited by 9
  9. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  10. 2025MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeXcited by 7
  11. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 7
  12. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Modelscited by 6
34 more
  1. 2025Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retrainingcited by 6
  2. 2026MoE Lens -- An Expert Is All You Needcited by 6
  3. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  4. 2025EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Modelscited by 5
  5. 2025BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebookcited by 4
  6. 2025MoPEQ: Mixture of Mixed Precision Quantized Expertscited by 4
  7. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Servingcited by 4
  8. 2026Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMscited by 3
  9. 2026Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decodingcited by 3
  10. 2026Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Expertscited by 3
  11. 2026OrderMoE: An expert similarity driven distributed edge MoE inferencecited by 3
  12. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  13. 2026Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Servingcited by 2
  14. 2025MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?cited by 1
  15. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inferencecited by 1
  16. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  17. 2026Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Studycited by 1
  18. 2025QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfigurationcited by 0
  19. 2025Graph-of-Causal Evolution: Challenging Chain-of-Model for Reasoningcited by 0
  20. 2026OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scalecited by 0
  21. 2026Networking-Aware Energy Efficiency in Agentic AI Inference: A Surveycited by 0
  22. 2026GEM: Graph-Enhanced Mixture-of-Experts with ReAct Agents for Dialogue State Trackingcited by 0
  23. 2026Fast MoE Inference via Predictive Prefetching and Expert Replicationcited by 0
  24. 2026MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptationcited by 0
  25. 2026CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Executioncited by 0
  26. 2026Benchmarking Edge Inference Strategies for Deep Learning Models in Industrial Machine Visioncited by 0
  27. 2026UMoE:Unlocking Every Expert in Domain-Specific Trainingcited by 0
  28. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0
  29. 2026cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMscited by 0
  30. 2026When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruningcited by 0
  31. 2026The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelismcited by 0
  32. 2026SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloadingcited by 0
  33. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Datacited by 0
  34. 2026Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing Distributed Deployment of Mixture-of-Experts Model Inference in Serverless Computing
  2. 2025HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference
  3. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
  4. 2023Revisiting Single-gated Mixtures of Experts
  5. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  6. 2026Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection
  7. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
  8. 2026Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization