mlsanityData to 7 Oct 2026

TrimMoE A communication aware and adaptive depth framework for distributed edge inference

arXiv 2608.00573 ↗First posted 1 Aug 2026cs.NI cs.CLcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Serving Mixture-of-Experts (MoE) large language models across distributed edge servers is bottlenecked by the cross-server expert transmission. The existing approaches mainly focus on how to reach a remote expert faster. However, in this paper, we instead consider whether a given layer, and the layers after it, need to be executed at all. To this end, a communication-aware adaptive-depth framework is proposed in this paper, termed TrimMoE, which couples layer skipping and confidence-based early exit with substitute execution and server-expert selection under a unified quality budget. Specifically, in the offline stage, TrimMoE freezes the backbone, trains the lightweight per-layer exit heads, calibrates the per-layer importance thresholds, and allocates the expert replicas by a skip/exit-aware redundancy benefit. In the online stage, a transition-aware look-ahead anticipates the token movement, so that the depth reduction targets the costliest transmissions, and besides, two feedback rules adapt the delay-quality weights and the exit threshold. Moreover, we prove that the substitution-and-skipping proxy degradation never exceeds the configured budget, and that the early exit is admitted only under a calibrated confidence gate. On a heterogeneous 10-server testbed with Switch-Base-8E, Qwen-MoE-A2.7B, and Mixtral-8x7B, TrimMoE reduces the average latency by up to 62.8%, lowers the cross-server traffic and the remote-execution ratio, and sustains high throughput under load, while keeping the task-quality degradation within a 2% bound.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 25 of its 46 references that are in the map. First posted on arXiv 1 Aug 2026.

Builds on

25 of 46 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026OrderMoE: An expert similarity driven distributed edge MoE inference19 Jul
  2. 2026PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetching31 Jan
  3. 2025SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inference9 Jul
  4. 2025MoE2: Optimizing Collaborative Inference for Edge Large Language Models16 Jan
  5. 2024WDMoE: Wireless Distributed Mixture of Experts for Large Language Models11 Nov
  6. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug

Other references · 19

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  6. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  7. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  8. 2022Confident Adaptive Language Modelingcited by 297
  9. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  10. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  11. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  12. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
7 more
  1. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  2. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 114
  3. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 98
  4. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  5. 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31
  6. 2025Jupiter: Fast and Resource-Efficient Collaborative Inference of Generative LLMs on Edge Devicescited by 15
  7. 2025Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networkscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement
  2. 2026EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding
  3. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate
  4. 2025Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
  5. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices
  6. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference
  7. 2024EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
  8. 2025PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networks