mlsanityData to 7 Oct 2026

HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference

arXiv 2608.00577 ↗First posted 1 Aug 2026cs.NI cs.CLcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) models have become a dominant architecture for large-scale AI services, yet deploying them over geo-distributed heterogeneous edge servers remains challenging. When the Top-k activated experts of a token are spread across multiple servers, the optimal routing depends jointly on cross-server link bandwidth, heterogeneous GPU computing capability, GPU-CPU expert loading delay, instantaneous queueing backlog, and replica-level quantization quality loss. Existing distributed inference and MoE serving methods address these factors separately and do not provide a unified framework for online multi-server collaborative routing. In this paper, we propose HetRoute, a heterogeneous-cost-aware collaborative routing framework for distributed edge MoE inference. HetRoute introduces a unified per-assignment cost model that explicitly captures four cost components: cross-server transmission, GPU-CPU offloading, GPU computation with queueing, and quantization-induced quality penalty. Guided by this model, the offline stage determines expert server placement, GPU-CPU residency, and replica precision through a routing-cost-coupled deployment algorithm, while the online stage routes the Top-k activated expert set as a whole by minimizing the bottleneck layer cost via exact enumeration or beam search. Theoretical analysis establishes fallback feasibility, a bound on the number of participating servers, per-layer optimality for small candidate domains, and online computational complexity. Trace-driven evaluation on three MoE models over a heterogeneous 10-server edge testbed shows that HetRoute reduces average inference latency by up to 59.0% and P99 latency by up to 58.0%, cuts cross-server traffic by up to 72.1%, and achieves 2.13x throughput improvement compared with representative baselines, while keeping quality degradation within the configured budget.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 29 of its 44 references that are in the map. First posted on arXiv 1 Aug 2026.

Builds on

29 of 44 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026OrderMoE: An expert similarity driven distributed edge MoE inference19 Jul
  2. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement18 Aug
  3. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving17 Apr
  4. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  5. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  6. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices28 Aug
  7. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  8. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun

Other references · 21

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  7. 2024Mixtral of Expertscited by 1,785
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2023A Survey on Multimodal Large Language Modelscited by 898
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  11. 2019Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computingcited by 225
  12. 2019Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computingcited by 150
9 more
  1. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  2. 2017Distributed Deep Neural Networks over the Cloud, the Edge and End Devicescited by 119
  3. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  4. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 98
  5. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  6. 2018JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Servicescited by 79
  7. 2020HAPI: Hardware-Aware Progressive Inferencecited by 30
  8. 2025Mixture-of-Experts for Distributed Edge Computing with Channel-Aware Gating Functioncited by 5
  9. 2023Mobility and Cost Aware Inference Accelerating Algorithm for Edge Intelligencecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TopoCompress: Topology Aware Token Compression Algorithm for Distributed Edge MoE Inference
  2. 2026Democratizing MoE inference on commodity GPUs with CoMoE
  3. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference
  4. 2025Stable-MoE: Lyapunov-based Token Routing for Distributed Mixture-of-Experts Training over Edge Networks
  5. 2026EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
  6. 2025MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs
  7. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
  8. 2026Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference