# WDMoE: Wireless Distributed Large Language Models with Mixture of Experts

arXiv: https://arxiv.org/abs/2405.03131 · HTML: https://mlsanity.com/paper/2405.03131/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 6 May 2024.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 17 of 17)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [SplitFed: When Federated Learning Meets Split Learning](https://mlsanity.com/paper/2004.12088/) (2020)
- [Drive Like a Human: Rethinking Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2307.07162/) (2023)
- [Split Learning over Wireless Networks: Parallel Design and Resource Management](https://mlsanity.com/paper/2204.08119/) (2022)
- [Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs](https://mlsanity.com/paper/2403.20041/) (2024)
- [Turn Waste into Worth: Rectifying Top-k Router of MoE](https://mlsanity.com/paper/2402.12399/) (2024)
- [An Edge-Cloud Collaboration Framework for Generative AI Service Provision with Synergetic Big Cloud Model and Small Edge Models](https://mlsanity.com/paper/2401.01666/) (2024)

## Built on by (most cited first, 18 of 18)

- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [WDMoE: Wireless Distributed Mixture of Experts for Large Language Models](https://mlsanity.com/paper/2411.06681/) (2024)
- [Leveraging Large Language Models for Integrated Satellite-Aerial-Terrestrial Networks: Recent Advances and Future Directions](https://mlsanity.com/paper/2407.04581/) (2024)
- [Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts](https://mlsanity.com/paper/2508.00234/) (2025)
- [DeepSeek-Inspired Exploration of RL-based LLMs and Synergy with Wireless Networks: A Survey](https://mlsanity.com/paper/2503.09956/) (2025)
- [Generative Semantic Communication: Architectures, Technologies, and Applications](https://mlsanity.com/paper/2412.08642/) (2024)
- [Mixture-of-Experts for Distributed Edge Computing with Channel-Aware Gating Function](https://mlsanity.com/paper/2504.00819/) (2025)
- [Optimal Expert Selection for Distributed Mixture-of-Experts at the Wireless Edge](https://mlsanity.com/paper/2503.13421/) (2025)
- [When MoE Meets Blockchain: A Trustworthy Distributed Framework of Large Models](https://mlsanity.com/paper/2509.12141/) (2025)
- [DiSCo: Device-Server Collaborative LLM-Based Text Streaming Services](https://mlsanity.com/paper/2502.11417/) (2025)
- [Agentic TinyML for Intent-aware Handover in 6G Wireless Networks](https://mlsanity.com/paper/2508.09147/) (2025)
- [CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systems](https://mlsanity.com/paper/2508.11287/) (2025)
- [FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computing](https://mlsanity.com/paper/2512.23070/) (2025)
- [OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference](https://mlsanity.com/paper/2512.03927/) (2025)
- [Empowering Large Language Models in Wireless Communication: A Novel Dataset and Fine-Tuning Framework](https://mlsanity.com/paper/2501.09631/) (2025)
- [AirMoE: Statistic-Augmented Over-the-Air MoE for Collaborative Intelligence](https://mlsanity.com/paper/2607.16562/) (2026)
- [WiSDoM: Wireless Sparse Decision Transformer with Mixture-of-Experts for Multi-Task Mobile Network Optimization](https://mlsanity.com/paper/2609.00284/) (2026)
- [GenomeOcean Anywhere: Private WebGPU Inference for Genome MoEs](https://mlsanity.com/paper/2609.35882/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have achieved significant success in various natural language processing tasks, but how wireless communications can support LLMs has not been extensively studied. In this paper, we propose a wireless distributed LLMs paradigm based on Mixture of Experts (MoE), named WDMoE, deploying LLMs collaboratively across edge servers of base station (BS) and mobile devices in the wireless communications system. Specifically, we decompose the MoE layer in LLMs by deploying the gating network and the preceding neural network layer at BS, while distributing the expert networks across the devices. This arrangement leverages the parallel capabilities of expert networks on distributed devices. Moreover, to overcome the instability of wireless communications, we design an expert selection policy by taking into account both the performance of the model and the end-to-end latency, which includes both transmission delay and inference delay. Evaluations conducted across various LLMs and multiple datasets demonstrate that WDMoE not only outperforms existing models, such as Llama 2 with 70 billion parameters, but also significantly reduces end-to-end latency.