mlsanityData to 14 Oct 2026

Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computing

arXiv 2512.18674 ↗First posted 21 Dec 2025cs.DC cs.AIcited by 2

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) has become a dominant architecture in large language models (LLMs) due to its ability to scale model capacity via sparse expert activation. Meanwhile, serverless computing, with its elasticity and pay-per-use billing, is well-suited for deploying MoEs with bursty workloads. However, the large number of experts in MoE models incurs high inference costs due to memory-intensive parameter caching. These costs are difficult to mitigate via simple model partitioning due to input-dependent expert activation. To address these issues, we propose Remoe, a heterogeneous MoE inference system tailored for serverless computing. Remoe assigns non-expert modules to GPUs and expert modules to CPUs, and further offloads infrequently activated experts to separate serverless functions to reduce memory overhead and enable parallel execution. We incorporate three key techniques: (1) a Similar Prompts Searching (SPS) algorithm to predict expert activation patterns based on semantic similarity of inputs; (2) a Main Model Pre-allocation (MMP) algorithm to ensure service-level objectives (SLOs) via worst-case memory estimation; and (3) a joint memory and replica optimization framework leveraging Lagrangian duality and the Longest Processing Time (LPT) algorithm. We implement Remoe on Kubernetes and evaluate it across multiple LLM benchmarks. Experimental results show that Remoe reduces inference cost by up to 57% and cold start latency by 47% compared to state-of-the-art baselines.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 17 of its 38 references that are in the map. First posted on arXiv 21 Dec 2025.

Builds on

17 of 38 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing9 Jun
  2. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference8 Apr
  3. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading7 Feb
  4. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection13 Nov
  5. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  6. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  7. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling23 Oct

Other references · 10

  1. 2016Pointer Sentinel Mixture Modelscited by 3,096
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  3. 2024DeepSeek-V3 Technical Reportcited by 811
  4. 2024Phi-4 Technical Reportcited by 711
  5. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  6. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  7. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 14
  8. 2024ENOVA: Autoscaling towards Cost-effective and Stable Serverless LLM Servingcited by 6
  9. 2024MOPAR: A Model Partitioning Framework for Deep Learning Inference Services on Serverless Platformscited by 2
  10. 2025Towards Resource-Efficient Serverless LLM Inference with SLINFERcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026MoEless: Efficient MoE LLM Serving via Serverless Computingcited by 1
  2. 2026OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloadingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference
  2. 2025Optimizing Distributed Deployment of Mixture-of-Experts Model Inference in Serverless Computing
  3. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  4. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  5. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  6. 2026BrownoutMoE: Structure-Aware Expert Grouping for Efficient and Accurate LLM Web-based Services
  7. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  8. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference