mlsanityData to 14 Oct 2026

MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

arXiv 2605.02960 ↗First posted 3 May 2026cs.LGcited by 2

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Production LLM workloads increasingly serve discriminative tasks, such as classification, recommendation, and verification, whose answers are read from the logits of a single prefill pass with no autoregressive decoding. Serving these prefill-only workloads on mixture-of-experts (MoE) models is bottlenecked not by compute but by the distributed execution required to fit the model: existing parallel strategies (tensor, expert, and pipeline parallelism) trade memory pressure for redundant computation, communication, and synchronization, severely degrading MoE prefill serving efficiency. We observe that these overheads stem from coupling expert placement with synchronous activation routing -- a design inherited from the decoding era. The long, compute-bound forward passes of large-batch prefill open a per-layer window wide enough to stream expert weights in the background, replacing per-layer activation AllToAll with asynchronous weight AllGather fully overlapped with computation. We propose MoE-Prefill, a prefill-only serving system whose backend, AsyncEP (Asynchronous Expert Parallelism), gathers experts by weight rather than routing them by activation, and whose frontend co-enforces a physically-derived saturation threshold through prefix-aware routing and true-FLOPs load tracking. On Qwen3-235B-A22B across four hardware/precision configurations, MoE-Prefill delivers 1.35-1.37x throughput over the strongest distributed baseline on real-world workloads and up to 1.59x on long-context synthetic workloads, sustaining 29.8-36.2% per-GPU model FLOPs utilization.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 60 of its 75 references that are in the map. First posted on arXiv 3 May 2026.

Builds on

60 of 75 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Models2 Oct
  2. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts26 Sep
  3. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  4. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference16 Oct
  5. 2024Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping30 Apr
  6. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  7. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference16 Jan
  8. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  9. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  10. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  11. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  12. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
1 more
  1. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 47

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2024Mixtral of Expertscited by 1,823
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
35 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  8. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  12. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  13. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 486
  14. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  15. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  16. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  17. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  18. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  19. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  20. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  21. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  22. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  23. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  24. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  25. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  26. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 135
  27. 2023Text Classification via Large Language Modelscited by 106
  28. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 82
  29. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 73
  30. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 32
  31. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  32. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  33. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  34. 2022STI: Turbocharge NLP Inference at the Edge via Elastic Pipeliningcited by 14
  35. 2025Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architecturescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUscited by 0
  2. 2026DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration
  2. 2026ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill
  3. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  4. 2026SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving
  5. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
  6. 2026ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels
  7. 2026MoEless: Efficient MoE LLM Serving via Serverless Computing
  8. 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models