# OrderMoE: An expert similarity driven distributed edge MoE inference

arXiv: https://arxiv.org/abs/2607.17154 · HTML: https://mlsanity.com/paper/2607.17154/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 19 Jul 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 23 of 23)

- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems](https://mlsanity.com/paper/2312.15234/) (2023)
- [A Review on Edge Large Language Models: Design, Execution, and Applications](https://mlsanity.com/paper/2410.11845/) (2024)
- [A Survey on Inference Optimization Techniques for Mixture of Experts Models](https://mlsanity.com/paper/2412.14219/) (2024)
- [AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference](https://mlsanity.com/paper/2408.10284/) (2024)
- [Layerwise Recurrent Router for Mixture-of-Experts](https://mlsanity.com/paper/2408.06793/) (2024)
- [Scalable Training of Mixture-of-Experts Models with Megatron Core](https://mlsanity.com/paper/2603.07685/) (2026)
- [Jupiter: Fast and Resource-Efficient Collaborative Inference of Generative LLMs on Edge Devices](https://mlsanity.com/paper/2504.08242/) (2025)
- [WDMoE: Wireless Distributed Mixture of Experts for Large Language Models](https://mlsanity.com/paper/2411.06681/) (2024)
- [Serving Long-Context LLMs at the Mobile Edge: Test-Time Reinforcement Learning-based Model Caching and Inference Offloading](https://mlsanity.com/paper/2501.14205/) (2025)
- [An Extensible Software Transport Layer for GPU Networking](https://mlsanity.com/paper/2504.17307/) (2025)
- [Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks](https://mlsanity.com/paper/2503.14882/) (2025)
- [SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inference](https://mlsanity.com/paper/2507.06567/) (2025)
- [Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement](https://mlsanity.com/paper/2508.12851/) (2025)
- [MoE2: Optimizing Collaborative Inference for Edge Large Language Models](https://mlsanity.com/paper/2501.09410/) (2025)
- [The MoE-Empowered Edge LLMs Deployment: Architecture, Challenges, and Opportunities](https://mlsanity.com/paper/2502.08381/) (2025)
- [Optimal Expert Selection for Distributed Mixture-of-Experts at the Wireless Edge](https://mlsanity.com/paper/2503.13421/) (2025)
- [PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetching](https://mlsanity.com/paper/2602.00509/) (2026)
- [Mobility and Cost Aware Inference Accelerating Algorithm for Edge Intelligence](https://mlsanity.com/paper/2312.16497/) (2023)
- [Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression](https://mlsanity.com/paper/2510.02345/) (2025)

## Built on by (most cited first, 3 of 3)

- [TrimMoE A communication aware and adaptive depth framework for distributed edge inference](https://mlsanity.com/paper/2608.00573/) (2026)
- [HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference](https://mlsanity.com/paper/2608.00577/) (2026)
- [TopoCompress: Topology Aware Token Compression Algorithm for Distributed Edge MoE Inference](https://mlsanity.com/paper/2609.26061/) (2026)

## Abstract (arXiv)

Although mixture-of-experts, MoE, models have been increasingly adopted to scale large language models with moderate computation cost, it remains challenging to deploy MoE inference over resource-constrained and bandwidth-limited edge infrastructures. Existing distributed MoE serving methods mainly rely on exact expert placement, caching, replication, or communication scheduling, while overlooking the functional similarity among experts, which provides an opportunity to reduce cross-server token transmission. Therefore, this paper introduces a similarity-aware expert allocation and distributed deployment framework, dubbed OrderMoE, which aims to accelerate edge MoE inference while balancing inference latency, communication overhead, server workload, and inference quality. OrderMoE first constructs an expert similarity model based on router-induced logits representations and partitions experts in each MoE layer into multiple similarity groups. Then, it develops a similarity-aware expert grouping and deployment strategy to improve local similarity coverage across edge servers. Since reducing remote expert invocation and preserving exact inference quality are conflicting objectives, OrderMoE further designs a quality-aware and trajectory-aware runtime server-expert selection algorithm to decide whether a token should invoke its remote target expert or use a feasible local substitute expert. Experimental results on a real distributed edge testbed show that OrderMoE significantly reduces average latency, tail latency, cross-server traffic, and remote expert invocation ratio, while introducing only small and controllable inference quality degradation.