mlsanityData to 14 Oct 2026

Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference

arXiv 2606.29982 ↗First posted 29 Jun 2026cs.DCcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) architectures enable language models to achieve unprecedented scale via sparse activation. However, their inference performance is often limited by data movement bottlenecks. Two coupled challenges exacerbate this limtation: (1) Importance-Agnostic Cost: Low-contribution experts incur nearly uniform memory and transfer costs, resulting in a low cost-to-benefit ratio and wasting critical bandwidth; (2) System-Level Imbalance: Multi-device deployments are universally bottlenecked by the slowest device, meaning that local reductions on one device may yield no improvement in end-to-end latency. We propose Cost-Aware Expert Execution (CAEE), a hardware-guided runtime framework that jointly optimizes for token-level expert importance and system-level execution cost. CAEE uses lightweight, calibrated cost models to estimate hardware overhead, selectively prunes low-importance, high-cost experts, and redistributes their contributions via a low-overhead compensation mechanism, avoiding extra data movement. Evaluations on the 671B DeepSeek-R1 model show that CAEE can reduce end-to-end inference latency by 8%-18% across diverse deployment settings, including expert offloading and on-device execution on multi-device systems, while maintaining a model accuracy drop of less than 1%.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 29 of its 35 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

29 of 35 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Models18 Dec
  2. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  3. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  4. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  5. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
  6. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  7. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  8. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2024Mixtral of Expertscited by 1,823
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  9. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  10. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  11. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  12. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
8 more
  1. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  2. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 56
  3. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 37
  4. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 37
  5. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  6. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectioncited by 28
  7. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16
  8. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Modelscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  2. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  3. 2025Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computing
  4. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  5. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  6. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
  7. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  8. 2026PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition