# Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving

arXiv: https://arxiv.org/abs/2509.17863 · HTML: https://mlsanity.com/paper/2509.17863/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 10 of its 28 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 10 of 10)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving](https://mlsanity.com/paper/2407.00079/) (2024)
- [Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines](https://mlsanity.com/paper/2107.06925/) (2021)
- [Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding](https://mlsanity.com/paper/2507.19427/) (2025)

## Built on by (most cited first, 13 of 13)

- [Janus: Disaggregating Attention and Experts for Scalable MoE Inference](https://mlsanity.com/paper/2512.13525/) (2025)
- [Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems](https://mlsanity.com/paper/2602.09721/) (2026)
- [Role-Based Fault Tolerance System for LLM RL Post-Training](https://mlsanity.com/paper/2512.22492/) (2025)
- [Making MoE-based LLM Inference Resilient with Tarragon](https://mlsanity.com/paper/2601.01310/) (2026)
- [Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start](https://mlsanity.com/paper/2604.06664/) (2026)
- [Surviving Partial Rank Failures in Wide Expert-Parallel MoE Inference](https://mlsanity.com/paper/2605.10670/) (2026)
- [From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models](https://mlsanity.com/paper/2511.02248/) (2025)
- [Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling](https://mlsanity.com/paper/2603.27624/) (2026)
- [FaaSMoE: A Serverless Framework for Multi-Tenant Mixture-of-Experts Serving](https://mlsanity.com/paper/2604.26881/) (2026)
- [NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decoding](https://mlsanity.com/paper/2605.21100/) (2026)
- [ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill](https://mlsanity.com/paper/2606.22541/) (2026)
- [AFD-Ledger: Deployment Provisioning for Attention--FFN Disaggregation](https://mlsanity.com/paper/2608.04502/) (2026)
- [OpScale: Operator-level Provisioning and Autoscaling for LLM Serving](https://mlsanity.com/paper/2608.13499/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) models challenge serving infrastructures with dynamic, sparse expert utilization, causing instability on conventional systems designed for dense architectures. We propose EaaS, a novel serving system to enable efficient, scalable, and robust MoE deployment. Our system disaggregates MoE modules into independent, stateless services. This design enables fine-grained resource scaling and provides inherent fault tolerance by decoupling compute units. The architecture is powered by a high-performance, CPU-free peer-to-peer communication library that ensures minimal overhead and high throughput. Experiments confirm EaaS's scalability and efficiency, achieving performance comparable to monolithic systems while providing robust fault tolerance and strong scalability. EaaS incurs less than a 2% throughput reduction under simulated hardware failures that would otherwise halt monolithic architectures. It further saves up to 37.5% of computing resources through dynamic fine-grained adaptation to serving traffic, demonstrating strong resilience for large-scale MoE deployment in production.