ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
Mixture-of-Experts (MoE) models have become the de facto standard for scaling large language models. To maintain computational efficiency, modern MoE serving systems typically employ a hybrid parallelism strategy, combining Data Parallelism (DP) for attention stages with Expert Parallelism (EP) for MoE stages. However, this design necessitates frequent global synchronization barriers between attention DP groups and experts. In online serving, significant variance in request arrival rates and sequence lengths inherently leads to DP imbalance, causing severe synchronization stalls that degrade Time-to-First-Token (TTFT) and system throughput. We present ASAP, an asynchronous inference system specifically designed to accelerate the prefill phase of MoE models. ASAP disaggregates the attention and MoE stages and implements a fully asynchronous execution pipeline. This is achieved through a suite of specialized asynchronous communication primitives and four coordinated optimizations across request scheduling and model execution, which collectively dismantle global synchronization barriers. We implement and evaluate ASAP on CloudMatrix384 super-nodes, demonstrating that it improves SLO-compliant prefill throughput by 90% compared to state-of-the-art synchronous serving solutions.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 37 of its 57 references that are in the map. First posted on arXiv 21 Jun 2026.
Builds on
37 of 57 references.
Closest ancestors · 5
Same direction, newest first
- 2026Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems10 Feb
- 2025Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving22 Sep
- 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding25 Jul
- 2025Serving Large Language Models on Huawei CloudMatrix38415 Jun
- 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
Other references · 32
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2019Generating Long Sequences with Sparse Transformerscited by 1,546
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
20 more
- 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
- 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
- 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
- 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
- 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 71
- 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
- 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 54
- 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
- 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
- 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 31
- 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
- 2020LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inferencecited by 22
- 2025Past-Future Scheduler for LLM Serving under SLA Guaranteescited by 20
- 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
- 2025DeepServe: Serverless Large Language Model Serving at Scalecited by 18
- 2025Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsitycited by 18
- 2025WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Trainingcited by 17
- 2025TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inferencecited by 14
- 2025UB-Mesh: a Hierarchically Localized nD-FullMesh Datacenter Network Architecturecited by 6
- 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Servingcited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
- 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
- 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference
- 2025ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Models
- 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
- 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inference
- 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference
- 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference