# Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference

arXiv: https://arxiv.org/abs/2606.29982 · HTML: https://mlsanity.com/paper/2606.29982/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models](https://mlsanity.com/paper/2402.14800/) (2024)
- [Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference](https://mlsanity.com/paper/2308.12066/) (2023)
- [Fast Inference of Mixture-of-Experts Language Models with Offloading](https://mlsanity.com/paper/2312.17238/) (2023)
- [MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs](https://mlsanity.com/paper/2411.11217/) (2024)
- [Task-Specific Expert Pruning for Sparse Mixture-of-Experts](https://mlsanity.com/paper/2206.00277/) (2022)
- [Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models](https://mlsanity.com/paper/2402.07033/) (2024)
- [MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router](https://mlsanity.com/paper/2410.12013/) (2024)
- [MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition](https://mlsanity.com/paper/2411.01016/) (2024)
- [Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models](https://mlsanity.com/paper/2405.14297/) (2024)
- [HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference](https://mlsanity.com/paper/2411.01433/) (2024)
- [A Survey on Inference Optimization Techniques for Mixture of Experts Models](https://mlsanity.com/paper/2412.14219/) (2024)
- [AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference](https://mlsanity.com/paper/2408.10284/) (2024)
- [ProMoE: Fast MoE-based LLM Serving using Proactive Caching](https://mlsanity.com/paper/2410.22134/) (2024)
- [Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference](https://mlsanity.com/paper/2303.06182/) (2023)
- [Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2407.09590/) (2024)
- [STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning](https://mlsanity.com/paper/2409.06211/) (2024)
- [XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection](https://mlsanity.com/paper/2403.18926/) (2024)
- [Adaptive Gating in Mixture-of-Experts based Language Models](https://mlsanity.com/paper/2310.07188/) (2023)
- [DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models](https://mlsanity.com/paper/2409.06669/) (2024)

## Built on by (most cited first, 1 of 1)

- [TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes](https://mlsanity.com/paper/2608.13057/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) architectures enable language models to achieve unprecedented scale via sparse activation. However, their inference performance is often limited by data movement bottlenecks. Two coupled challenges exacerbate this limtation: (1) Importance-Agnostic Cost: Low-contribution experts incur nearly uniform memory and transfer costs, resulting in a low cost-to-benefit ratio and wasting critical bandwidth; (2) System-Level Imbalance: Multi-device deployments are universally bottlenecked by the slowest device, meaning that local reductions on one device may yield no improvement in end-to-end latency. We propose Cost-Aware Expert Execution (CAEE), a hardware-guided runtime framework that jointly optimizes for token-level expert importance and system-level execution cost. CAEE uses lightweight, calibrated cost models to estimate hardware overhead, selectively prunes low-importance, high-cost experts, and redistributes their contributions via a low-overhead compensation mechanism, avoiding extra data movement. Evaluations on the 671B DeepSeek-R1 model show that CAEE can reduce end-to-end inference latency by 8%-18% across diverse deployment settings, including expert offloading and on-device execution on multi-device systems, while maintaining a model accuracy drop of less than 1%.