mlsanityData to 7 Oct 2026

MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism

arXiv 2504.02263 ↗First posted 3 Apr 2025cs.DC cs.LGcited by 31

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) showcases tremendous potential to scale large language models (LLMs) with enhanced performance and reduced computational complexity. However, its sparsely activated architecture shifts feed-forward networks (FFNs) from being compute-intensive to memory-intensive during inference, leading to substantially lower GPU utilization and increased operational costs. We present MegaScale-Infer, an efficient and cost-effective system for serving large-scale MoE models. MegaScale-Infer disaggregates attention and FFN modules within each model layer, enabling independent scaling, tailored parallelism strategies, and heterogeneous deployment for both modules. To fully exploit disaggregation in the presence of MoE's sparsity, MegaScale-Infer introduces ping-pong pipeline parallelism, which partitions a request batch into micro-batches and shuttles them between attention and FFNs for inference. Combined with distinct model parallelism for each module, MegaScale-Infer effectively hides communication overhead and maximizes GPU utilization. To adapt to disaggregated attention and FFN modules and minimize data transmission overhead (e.g., token dispatch), MegaScale-Infer provides a high-performance M2N communication library that eliminates unnecessary GPU-to-CPU data copies, group initialization overhead, and GPU synchronization. Experimental results indicate that MegaScale-Infer achieves up to 1.90x higher per-GPU throughput than state-of-the-art solutions.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 35 of its 74 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

35 of 74 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  3. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  4. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
18 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  3. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  4. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  5. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  6. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  7. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  8. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  9. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  10. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 53
  11. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  12. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  13. 2021Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloadscited by 40
  14. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  15. 2020Synthesizing Optimal Collective Algorithmscited by 30
  16. 2023Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problemcited by 11
  17. 2022Not All GPUs Are Created Equal: Characterizing Variability in Large-Scale, Accelerator-Rich Systemscited by 9
  18. 2020GPA: A GPU Performance Advisor Based on Instruction Samplingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 19.

31 papers.

Sort:

By area:

  1. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  2. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 8
  3. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  4. 2025Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inferencecited by 5
  5. 2026PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetchingcited by 4
  6. 2026Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systemscited by 4
  7. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Trainingcited by 4
  8. 2026Frontier: Towards Comprehensive and Accurate LLM Inference Simulationcited by 4
  9. 2025Frontier: Simulating the Next Generation of LLM Inference Systemscited by 2
  10. 2025Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Frameworkcited by 2
  11. 2025Understanding and Improving Communication Performance in Multi-node LLM Inferencecited by 2
  12. 2025Role-Based Fault Tolerance System for LLM RL Post-Trainingcited by 2
19 more
  1. 2026Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloadscited by 2
  2. 2026CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Servingcited by 2
  3. 2026How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Servingcited by 2
  4. 2025DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstructioncited by 1
  5. 2025MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inferencecited by 1
  6. 2025CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancingcited by 1
  7. 2026MixServe: An Automatic Distributed Serving System for MoE Models with Hybrid Parallelism Based on Fused Communication Algorithmcited by 1
  8. 2026HetCCL: Accelerating LLM Training with Heterogeneous GPUscited by 1
  9. 2026Surviving Partial Rank Failures in Wide Expert-Parallel MoE Inferencecited by 1
  10. 2026PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantizationcited by 1
  11. 2025From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Modelscited by 0
  12. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0
  13. 2026AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Servingcited by 0
  14. 2026DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelismcited by 0
  15. 2026Sieve: Dynamic Expert-Aware PIM Acceleration for Evolving Mixture-of-Experts Modelscited by 0
  16. 2026KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decodingcited by 0
  17. 2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineeringcited by 0
  18. 2026OpScale: Operator-level Provisioning and Autoscaling for LLM Servingcited by 0
  19. 2026Q-First: Most of Attention Needs Only the Query in Disaggregated LLM Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
  2. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
  3. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inference
  4. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  5. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  6. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  7. 2026ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill
  8. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism