mlsanityData to 7 Oct 2026

MoEless: Efficient MoE LLM Serving via Serverless Computing

arXiv 2603.06350 ↗First posted 6 Mar 2026cs.DC cs.AI cs.LGcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) increasingly adopt Mixture-of-Experts (MoE) architectures to scale efficiently under stringent resource constraints. However, MoE's sparse activation causes severe expert load imbalance, where a few experts become stragglers while others remain underutilized, leading to inflated inference latency and cost. Existing solutions assume static, serverful model deployments, limiting expert elasticity and often incurring costly expert swapping or degraded output quality. We present MoEless, an efficient serverless MoE serving framework that mitigates expert load imbalance via elastic expert execution. MoEless leverages lightweight, layer-aware predictors to estimate incoming expert load distributions and proactively identify stragglers. We design optimized scaling and placement strategies to improve function locality, GPU utilization, and cross-expert load balance. MoEless is prototyped on top of Megatron-LM and deployed on an eight-GPU testbed. Experiments with open-source MoE models and real-world workloads show that MoEless reduces inference latency by 43% and inference cost by 84% compared to state-of-the-art solutions.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 35 of its 84 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

35 of 84 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computing21 Dec
  2. 2025MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing9 Jun
  3. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core21 Apr
  4. 2025Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts7 Mar
  5. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts27 Feb
  6. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading7 Feb
  7. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference16 Dec
  8. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  9. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts7 Apr
  10. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  11. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  12. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
1 more
  1. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2024Mixtral of Expertscited by 1,785
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  11. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  12. 2024DeepSeek-V3 Technical Reportcited by 724
10 more
  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  3. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  5. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  6. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  7. 2025DeepServe: Serverless Large Language Model Serving at Scalecited by 17
  8. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 12
  9. 2024Formation of Representations in Neural Networkscited by 10
  10. 2025ServerlessLoRA: Enabling Low-Latency Serverless Multi-LoRA Servingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inferencecited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026BrownoutMoE: Structure-Aware Expert Grouping for Efficient and Accurate LLM Web-based Services
  2. 2025Faster MoE LLM Inference for Extremely Large Models
  3. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  4. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  5. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  6. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference
  7. 2025ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Models
  8. 2026FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs