# Theory of Mixture-of-Experts for Mobile Edge Computing

arXiv: https://arxiv.org/abs/2412.15690 · HTML: https://mlsanity.com/paper/2412.15690/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 23 of 23)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Characterizing Implicit Bias in Terms of Optimization Geometry](https://mlsanity.com/paper/1802.08246/) (2018)
- [MegaBlocks: Efficient Sparse Training with Mixture-of-Experts](https://mlsanity.com/paper/2211.15841/) (2022)
- [A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix](https://mlsanity.com/paper/2010.04003/) (2020)
- [How catastrophic can catastrophic forgetting be in linear regression?](https://mlsanity.com/paper/2205.09588/) (2022)
- [A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity](https://mlsanity.com/paper/2302.06015/) (2023)
- [Theory on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2302.05836/) (2023)
- [Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer](https://mlsanity.com/paper/2211.00789/) (2022)
- [Theory on Mixture-of-Experts in Continual Learning](https://mlsanity.com/paper/2406.16437/) (2024)
- [A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training](https://mlsanity.com/paper/2303.06318/) (2023)
- [CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One](https://mlsanity.com/paper/2207.06543/) (2022)
- [MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services](https://mlsanity.com/paper/2205.10034/) (2022)
- [LocMoE: A Low-Overhead MoE for Large Language Model Training](https://mlsanity.com/paper/2401.13920/) (2024)
- [Toward Scalable Generative AI via Mixture of Experts in Mobile Edge Networks](https://mlsanity.com/paper/2402.06942/) (2024)
- [Follow Me at the Edge: Mobility-Aware Dynamic Service Placement for Mobile Edge Computing](https://mlsanity.com/paper/1809.05239/) (2018)
- [Pricing-Driven Service Caching and Task Offloading in Mobile Edge Computing](https://mlsanity.com/paper/2011.02154/) (2020)

## Built on by (most cited first, 5 of 5)

- [A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications](https://mlsanity.com/paper/2503.07137/) (2025)
- [MoE2: Optimizing Collaborative Inference for Edge Large Language Models](https://mlsanity.com/paper/2501.09410/) (2025)
- [Plasticity-Aware Mixture of Experts for Learning Under QoE Shifts in Adaptive Video Streaming](https://mlsanity.com/paper/2504.09906/) (2025)
- [Don't Collapse Your Features: Why CenterLoss Hurts OOD Detection and Multi-Scale Mahalanobis Wins](https://mlsanity.com/paper/2605.21493/) (2026)
- [AirMoE: Statistic-Augmented Over-the-Air MoE for Collaborative Intelligence](https://mlsanity.com/paper/2607.16562/) (2026)

## Abstract (arXiv)

In mobile edge computing (MEC) networks, mobile users generate diverse machine learning tasks dynamically over time. These tasks are typically offloaded to the nearest available edge server, by considering communication and computational efficiency. However, its operation does not ensure that each server specializes in a specific type of tasks and leads to severe overfitting or catastrophic forgetting of previous tasks. To improve the continual learning (CL) performance of online tasks, we are the first to introduce mixture-of-experts (MoE) theory in MEC networks and save MEC operation from the increasing generalization error over time. Our MoE theory treats each MEC server as an expert and dynamically adapts to changes in server availability by considering data transfer and computation time. Unlike existing MoE models designed for offline tasks, ours is tailored for handling continuous streams of tasks in the MEC environment. We introduce an adaptive gating network in MEC to adaptively identify and route newly arrived tasks of unknown data distributions to available experts, enabling each expert to specialize in a specific type of tasks upon convergence. We derived the minimum number of experts required to match each task with a specialized, available expert. Our MoE approach consistently reduces the overall generalization error over time, unlike the traditional MEC approach. Interestingly, when the number of experts is sufficient to ensure convergence, adding more experts delays the convergence time and worsens the generalization error. Finally, we perform extensive experiments on real datasets in deep neural networks (DNNs) to verify our theoretical results.