mlsanityData to 14 Oct 2026

MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching

arXiv 2503.09716 ↗First posted 12 Mar 2025cs.DC cs.LGcited by 6

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

This paper presents MoE-Gen, a high-throughput MoE inference system optimized for single-GPU execution. Existing inference systems rely on model-based or continuous batching strategies, originally designed for interactive inference, which result in excessively small batches for MoE's key modules-attention and expert modules-leading to poor throughput. To address this, we introduce module-based batching, which accumulates tokens in host memory and dynamically launches large batches on GPUs to maximize utilization. Additionally, we optimize the choice of batch sizes for each module in an MoE to fully overlap GPU computation and communication, maximizing throughput. Evaluation demonstrates that MoE-Gen achieves 8-31x higher throughput compared to state-of-the-art systems employing model-based batching (FlexGen, MoE-Lightning, DeepSpeed), and offers even greater throughput improvements over continuous batching systems (e.g., vLLM and Ollama) on popular MoE models (DeepSeek and Mixtral) across offline inference tasks. MoE-Gen's source code is publicly available at https://github.com/EfficientMoE/MoE-Gen

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 21 of its 38 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

21 of 38 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  2. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  3. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  4. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  5. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  6. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct

Other references · 15

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  10. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  11. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  12. 2022Can Foundation Models Wrangle Your Data?cited by 175
3 more
  1. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  2. 2024Contextual Feature Extraction Hierarchies Converge in Large Language Models and the Braincited by 32
  3. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Serverscited by 8
  2. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  3. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraintscited by 3
  4. 2025HybridServe: Efficient Serving of Large AI Models with Confidence-Based Cascade Routingcited by 1
  5. 2025MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?cited by 1
  6. 2026TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloadingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
  2. 2025Accelerating MoE Model Inference with Expert Sharding
  3. 2026CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Execution
  4. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  5. 2025Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference
  6. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
  7. 2025Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems
  8. 2026GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems