mlsanityData to 14 Oct 2026

WDMoE: Wireless Distributed Large Language Models with Mixture of Experts

arXiv 2405.03131 ↗First posted 6 May 2024cs.IT cs.AI cs.LGcited by 18

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Large Language Models (LLMs) have achieved significant success in various natural language processing tasks, but how wireless communications can support LLMs has not been extensively studied. In this paper, we propose a wireless distributed LLMs paradigm based on Mixture of Experts (MoE), named WDMoE, deploying LLMs collaboratively across edge servers of base station (BS) and mobile devices in the wireless communications system. Specifically, we decompose the MoE layer in LLMs by deploying the gating network and the preceding neural network layer at BS, while distributing the expert networks across the devices. This arrangement leverages the parallel capabilities of expert networks on distributed devices. Moreover, to overcome the instability of wireless communications, we design an expert selection policy by taking into account both the performance of the model and the end-to-end latency, which includes both transmission delay and inference delay. Evaluations conducted across various LLMs and multiple datasets demonstrate that WDMoE not only outperforms existing models, such as Llama 2 with 70 billion parameters, but also significantly reduces end-to-end latency.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 17 of its 25 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

17 of 25 references.

References · 17

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2024Mixtral of Expertscited by 1,823
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  11. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
  12. 2020SplitFed: When Federated Learning Meets Split Learningcited by 310
5 more
  1. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 160
  2. 2022Split Learning over Wireless Networks: Parallel Design and Resource Managementcited by 75
  3. 2024Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUscited by 17
  4. 2024Turn Waste into Worth: Rectifying Top-k Router of MoEcited by 11
  5. 2024An Edge-Cloud Collaboration Framework for Generative AI Service Provision with Synergetic Big Cloud Model and Small Edge Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 11; 2026: 3.

18 papers.

Sort:

By area:

  1. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  2. 2024WDMoE: Wireless Distributed Mixture of Experts for Large Language Modelscited by 12
  3. 2024Leveraging Large Language Models for Integrated Satellite-Aerial-Terrestrial Networks: Recent Advances and Future Directionscited by 10
  4. 2025Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Expertscited by 7
  5. 2025DeepSeek-Inspired Exploration of RL-based LLMs and Synergy with Wireless Networks: A Surveycited by 6
  6. 2024Generative Semantic Communication: Architectures, Technologies, and Applicationscited by 5
  7. 2025Mixture-of-Experts for Distributed Edge Computing with Channel-Aware Gating Functioncited by 5
  8. 2025Optimal Expert Selection for Distributed Mixture-of-Experts at the Wireless Edgecited by 4
  9. 2025When MoE Meets Blockchain: A Trustworthy Distributed Framework of Large Modelscited by 3
  10. 2025DiSCo: Device-Server Collaborative LLM-Based Text Streaming Servicescited by 2
  11. 2025Agentic TinyML for Intent-aware Handover in 6G Wireless Networkscited by 2
  12. 2025CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systemscited by 2
6 more
  1. 2025FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computingcited by 2
  2. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inferencecited by 1
  3. 2025Empowering Large Language Models in Wireless Communication: A Novel Dataset and Fine-Tuning Frameworkcited by 0
  4. 2026AirMoE: Statistic-Augmented Over-the-Air MoE for Collaborative Intelligencecited by 0
  5. 2026WiSDoM: Wireless Sparse Decision Transformer with Mixture-of-Experts for Multi-Task Mobile Network Optimizationcited by 0
  6. 2026GenomeOcean Anywhere: Private WebGPU Inference for Genome MoEscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LocMoE: A Low-Overhead MoE for Large Language Model Training
  2. 2026Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
  3. 2024GMoE: Empowering LLMs Fine-Tuning via MoE Graph Collaboration
  4. 2024WirelessLLM: Empowering Large Language Models Towards Wireless Intelligence
  5. 2025Large Language Models for Wireless Communications: From Adaptation to Autonomy
  6. 2026FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
  7. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design
  8. 2024Performance Characterization of Expert Router for Scalable LLM Inference