mlsanityData to 13 Oct 2026

Optimizing Distributed Deployment of Mixture-of-Experts Model Inference in Serverless Computing

arXiv 2501.05313 ↗First posted 9 Jan 2025cs.DC cs.LGcited by 3

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

With the advancement of serverless computing, running machine learning (ML) inference services over a serverless platform has been advocated, given its labor-free scalability and cost effectiveness. Mixture-of-Experts (MoE) models have been a dominant type of model architectures to enable large models nowadays, with parallel expert networks. Serving large MoE models on serverless computing is potentially beneficial, but has been underexplored due to substantial challenges in handling the skewed expert popularity and scatter-gather communication bottleneck in MoE model execution, for cost-efficient serverless MoE deployment and performance guarantee. We study optimized MoE model deployment and distributed inference serving on a serverless platform, that effectively predict expert selection, pipeline communication with model execution, and minimize the overall billed cost of serving MoE models. Especially, we propose a Bayesian optimization framework with multi-dimensional epsilon-greedy search to learn expert selections and optimal MoE deployment achieving optimal billed cost, including: 1) a Bayesian decision-making method for predicting expert popularity; 2) flexibly pipelined scatter-gather communication; and 3) an optimal model deployment algorithm for distributed MoE serving. Extensive experiments on AWS Lambda show that our designs reduce the billed cost of all MoE layers by at least 75.67% compared to CPU clusters while maintaining satisfactory inference throughput. As compared to LambdaML in serverless computing, our designs achieves 43.41% lower cost with a throughput decrease of at most 18.76%.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 13 of its 40 references that are in the map. First posted on arXiv 9 Jan 2025.

Builds on

13 of 40 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing25 Apr
  2. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  4. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 9

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 896
  4. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  5. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  6. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 53
  7. 2021A Holistic View on Resource Management in Serverless Computing Environments: Taxonomy and Future Directionscited by 5
  8. 2020ε-shotgun: ε-greedy Batch Bayesian Optimisationcited by 3
  9. 2024Distributed Inference Performance Optimization for LLMs on CPUscited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  2. 2026RLHFless: Serverless Computing for Efficient RLHFcited by 2
  3. 2026FaaSMoE: A Serverless Framework for Multi-Tenant Mixture-of-Experts Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Models
  2. 2025Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computing
  3. 2026MoEless: Efficient MoE LLM Serving via Serverless Computing
  4. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  5. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference
  6. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference
  7. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
  8. 2026EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference