mlsanityData to 14 Oct 2026

EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference

arXiv 2410.12247 ↗First posted 16 Oct 2024cs.CL cs.DCcited by 6

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The Mixture-of-Experts (MoE) model has emerged as a prominent architecture in the field of Large Language Models (LLMs), providing a better balance between model performance and computational efficiency. However the General Matrix Multiply (GEMM) operations and large parameters introduce challenges related to computational efficiency and communication overhead, which become throughput bottlenecks during inference. Applying a single parallelism strategy like EP, DP, TP or a straightforward combination of them to MoE usually achieves sub-optimal inference throughput. This paper introduces EPS-MoE, a novel expert pipeline scheduler for MoE that surpasses the existing parallelism schemes. Our approach optimizes the computation of MoE FeedForward Network (FFN) modules by dynamically selecting the best kernel implementation of GroupGemm and DenseGemm for different loads and adaptively overlapping these computations with communication, leading to a substantial increase in throughput. Our experimental results demonstrate at most 52.4% improvement in prefill throughput compared to existing parallel inference methods. Specifically, our method accelerated the highly optimized DeepSeekV2 model from a claimed 100K tokens per second to at least 120K tokens per second.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 27 of its 31 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

27 of 31 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts7 Apr
  2. 2024Scattered Mixture-of-Experts Implementation13 Mar
  3. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training11 Mar
  4. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System28 Mar
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  6. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 21

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  9. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  10. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  11. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  12. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
9 more
  1. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  2. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  3. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90
  4. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 61
  5. 2020Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systemscited by 60
  6. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 53
  7. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 29
  8. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 27
  9. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 3.

6 papers.

  1. 2025LongCat-Flash Technical Reportcited by 58
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  3. 2026Scaling Embeddings Outperforms Scaling Experts in Language Modelscited by 15
  4. 2025Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Schedulingcited by 9
  5. 2026MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Servingcited by 2
  6. 2026HDA-MoE: Hybrid Parallelism and Dynamic, Adaptive Scheduling for Mixture-of-Experts with 3D Near-Memory Processingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  2. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism
  3. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
  4. 2025Faster MoE LLM Inference for Extremely Large Models
  5. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
  6. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
  7. 2025Accelerating MoE Model Inference with Expert Sharding
  8. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference