mlsanityData to 7 Oct 2026

MoE2: Optimizing Collaborative Inference for Edge Large Language Models

arXiv 2501.09410 ↗First posted 16 Jan 2025cs.NI cs.AI cs.LGcited by 6

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing tasks. Exploiting the heterogeneous capabilities of edge LLMs is crucial for diverse emerging applications, as it enables greater cost-effectiveness and reduced latency. In this work, we introduce textitMixture-of-Edge-Experts (MoE2), a novel collaborative inference framework for edge LLMs. We formulate the joint gating and expert selection problem to optimize inference performance under energy and latency constraints. Unlike conventional MoE problems, LLM expert selection is significantly more challenging due to the combinatorial nature and the heterogeneity of edge LLMs across various attributes. To this end, we propose a two-level expert selection mechanism through which we uncover an optimality-preserving property of gating parameters across expert selections. This property enables the decomposition of the training and selection processes, significantly reducing complexity. Furthermore, we leverage the objective's monotonicity and design a discrete monotonic optimization algorithm for optimal expert selection. We implement edge servers with NVIDIA Jetson AGX Orins and NVIDIA RTX 4090 GPUs, and perform extensive experiments. Our results validate that performance improvements of various LLM models and show that our MoE2 method can achieve optimal trade-offs among different delay and energy budgets, and outperforms baselines under various system resource constraints.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 56 of its 90 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

56 of 90 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices28 Aug

Other references · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
38 more
  1. 2023PaLM 2 Technical Reportcited by 1,154
  2. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 680
  3. 2023A decoder-only foundation model for time-series forecastingcited by 650
  4. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  6. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  7. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  8. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  9. 2022Confident Adaptive Language Modelingcited by 297
  10. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  11. 2023Reward Design with Language Modelscited by 257
  12. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  13. 2024More Agents Is All You Needcited by 174
  14. 2022PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecastingcited by 170
  15. 2019Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computingcited by 150
  16. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  17. 2023LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasterscited by 127
  18. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  19. 2022Benchmarking Large Language Models for Automated Verilog RTL Code Generationcited by 112
  20. 2021Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approachcited by 105
  21. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  22. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  23. 2024NetLLM: Adapting Large Language Models for Networkingcited by 76
  24. 2022ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classificationcited by 66
  25. 2022SecureBERT: A Domain-Specific Language Model for Cybersecuritycited by 60
  26. 2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58
  27. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 55
  28. 2023OptiMUS: Optimization Modeling Using MIP Solvers and large language modelscited by 39
  29. 2023Revolutionizing Cyber Threat Detection with Large Language Models: A privacy-preserving BERT-based Lightweight Model for IoT/IIoT Devicescited by 38
  30. 2023Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Roboticscited by 35
  31. 2023Towards Optimizing with Large Language Modelscited by 34
  32. 2023Understanding Telecom Language Through Large Language Modelscited by 32
  33. 2023What do LLMs need to Synthesize Correct Router Configurations?cited by 29
  34. 2023Large Language Model for Multi-objective Evolutionary Optimizationcited by 29
  35. 2023Diagnosing Infeasible Optimization Problems Using Large Language Modelscited by 27
  36. 2023The Power of Large Language Models for Wireless Communication System Development: A Case Study on FPGA Platformscited by 20
  37. 2023Enhancing Network Management Using Code Generated by Large Language Modelscited by 20
  38. 2024LocMoE: A Low-Overhead MoE for Large Language Model Trainingcited by 18

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  2. 2026OrderMoE: An expert similarity driven distributed edge MoE inferencecited by 3
  3. 2025EC2MoE: Adaptive End-Cloud Pipeline Collaboration Enabling Scalable Mixture-of-Experts Inferencecited by 2
  4. 2026Knowledge boundary probing and demand-guided intervention for LLM-based power system code generationcited by 1
  5. 2025Faster, Smaller, and Smarter: Task-Aware Expert Merging for Online MoE Inferencecited by 0
  6. 2026TrimMoE A communication aware and adaptive depth framework for distributed edge inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate
  2. 2025SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inference
  3. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement
  4. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  5. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  6. 2025Efficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborative
  7. 2024EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
  8. 2025Mixture of Experts in Large Language Models