# Optimizing Distributed Deployment of Mixture-of-Experts Model Inference in Serverless Computing

arXiv: https://arxiv.org/abs/2501.05313 · HTML: https://mlsanity.com/paper/2501.05313/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 13 of its 40 references that are in the map.
- First posted on arXiv 9 Jan 2025.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [FastMoE: A Fast Mixture-of-Expert Training System](https://mlsanity.com/paper/2103.13262/) (2021)
- [Accelerating Distributed MoE Training and Inference with Lina](https://mlsanity.com/paper/2210.17223/) (2022)
- [ServerlessLLM: Low-Latency Serverless Inference for Large Language Models](https://mlsanity.com/paper/2401.14351/) (2024)
- [bert2BERT: Towards Reusable Pretrained Language Models](https://mlsanity.com/paper/2110.07143/) (2021)
- [FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement](https://mlsanity.com/paper/2304.03946/) (2023)
- [Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing](https://mlsanity.com/paper/2404.16914/) (2024)
- [A Holistic View on Resource Management in Serverless Computing Environments: Taxonomy and Future Directions](https://mlsanity.com/paper/2105.11592/) (2021)
- [ε-shotgun: ε-greedy Batch Bayesian Optimisation](https://mlsanity.com/paper/2002.01873/) (2020)
- [Distributed Inference Performance Optimization for LLMs on CPUs](https://mlsanity.com/paper/2407.00029/) (2024)

## Built on by (most cited first, 3 of 3)

- [Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading](https://mlsanity.com/paper/2502.05370/) (2025)
- [RLHFless: Serverless Computing for Efficient RLHF](https://mlsanity.com/paper/2602.22718/) (2026)
- [FaaSMoE: A Serverless Framework for Multi-Tenant Mixture-of-Experts Serving](https://mlsanity.com/paper/2604.26881/) (2026)

## Abstract (arXiv)

With the advancement of serverless computing, running machine learning (ML) inference services over a serverless platform has been advocated, given its labor-free scalability and cost effectiveness. Mixture-of-Experts (MoE) models have been a dominant type of model architectures to enable large models nowadays, with parallel expert networks. Serving large MoE models on serverless computing is potentially beneficial, but has been underexplored due to substantial challenges in handling the skewed expert popularity and scatter-gather communication bottleneck in MoE model execution, for cost-efficient serverless MoE deployment and performance guarantee. We study optimized MoE model deployment and distributed inference serving on a serverless platform, that effectively predict expert selection, pipeline communication with model execution, and minimize the overall billed cost of serving MoE models. Especially, we propose a Bayesian optimization framework with multi-dimensional epsilon-greedy search to learn expert selections and optimal MoE deployment achieving optimal billed cost, including: 1) a Bayesian decision-making method for predicting expert popularity; 2) flexibly pipelined scatter-gather communication; and 3) an optimal model deployment algorithm for distributed MoE serving. Extensive experiments on AWS Lambda show that our designs reduce the billed cost of all MoE layers by at least 75.67% compared to CPU clusters while maintaining satisfactory inference throughput. As compared to LambdaML in serverless computing, our designs achieves 43.41% lower cost with a throughput decrease of at most 18.76%.