# CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Serving

arXiv: https://arxiv.org/abs/2603.28768 · HTML: https://mlsanity.com/paper/2603.28768/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 21 of its 73 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 21 of 21)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [ProMoE: Fast MoE-based LLM Serving using Proactive Caching](https://mlsanity.com/paper/2410.22134/) (2024)
- [MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism](https://mlsanity.com/paper/2504.02263/) (2025)
- [MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production](https://mlsanity.com/paper/2505.11432/) (2025)
- [ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production](https://mlsanity.com/paper/2505.09999/) (2025)
- [MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing](https://mlsanity.com/paper/2502.06643/) (2025)
- [ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling](https://mlsanity.com/paper/2410.17954/) (2024)
- [GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference](https://mlsanity.com/paper/2509.25041/) (2025)
- [eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference](https://mlsanity.com/paper/2503.06823/) (2025)
- [LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers](https://mlsanity.com/paper/2509.23638/) (2025)
- [SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling](https://mlsanity.com/paper/2504.19925/) (2025)
- [DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurance](https://mlsanity.com/paper/2509.07379/) (2025)
- [Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference](https://mlsanity.com/paper/2505.13345/) (2025)
- [MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing](https://mlsanity.com/paper/2506.07366/) (2025)

## Built on by (most cited first, 2 of 2)

- [GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems](https://mlsanity.com/paper/2605.19945/) (2026)
- [HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference](https://mlsanity.com/paper/2607.11586/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) has recently emerged as the mainstream architecture for efficiently scaling large language models while maintaining near-constant computational cost. Expert parallelism distributes parameters by partitioning experts across devices, but this introduces token-level load imbalance during inference. Expert replication is a widely adopted load-balancing technique in serving frameworks that alleviates load imbalance in large-scale deployments by replicating experts with high loads. In this work, we demonstrate that existing replication schemes often over-replicate, with many replicas providing marginal improvement. Replicas consume substantial GPU memory, which may lead to resource contention and throughput degradation. We present CRAFT, an efficient expert replication framework that maximizes load balance under a given memory budget by performing fine-grained, per-layer replication based on the estimated replication benefit. CRAFT can be seamlessly integrated into existing serving frameworks without any additional training or model changes. Our evaluation shows that CRAFT increases end-to-end serving throughput by 1.14× on average (up to 1.2×) over existing replication techniques in large-scale deployments with models ranging from hundreds of billions to a trillion parameters.