mlsanityData to 7 Oct 2026

EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices

arXiv 2308.14352 ↗First posted 28 Aug 2023cs.LG cs.AI cs.CLcited by 65

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Large language models (LLMs) such as GPTs and Mixtral-8x7B have revolutionized machine intelligence due to their exceptional abilities in generic ML tasks. Transiting LLMs from datacenters to edge devices brings benefits like better privacy and availability, but is challenged by their massive parameter size and thus unbearable runtime costs. To this end, we present EdgeMoE, an on-device inference engine for mixture-of-expert (MoE) LLMs -- a popular form of sparse LLM that scales its parameter size with almost constant computing complexity. EdgeMoE achieves both memory- and compute-efficiency by partitioning the model into the storage hierarchy: non-expert weights are held in device memory; while expert weights are held on external storage and fetched to memory only when activated. This design is motivated by a key observation that expert weights are bulky but infrequently used due to sparse activation. To further reduce the expert I/O swapping overhead, EdgeMoE incorporates two novel techniques: (1) expert-wise bitwidth adaptation that reduces the expert sizes with tolerable accuracy loss; (2) expert preloading that predicts the activated experts ahead of time and preloads it with the compute-I/O pipeline. On popular MoE LLMs and edge devices, EdgeMoE showcase significant memory savings and speedup over competitive baselines. The code is available at https://github.com/UbiquitousLearning/mllm.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 50 of its 94 references that are in the map. First posted on arXiv 28 Aug 2023.

Builds on

50 of 94 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
38 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  7. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  8. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  9. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  10. 2023Adapting Language Models to Compress Contextscited by 350
  11. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  12. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  13. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  14. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  15. 2021Hash Layers For Large Sparse Modelscited by 236
  16. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  17. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 174
  18. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  19. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  20. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  21. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  22. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  23. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  24. 2020SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloudcited by 109
  25. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  26. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 103
  27. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  28. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  29. 2020PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruningcited by 75
  30. 2017DeepCache: Principled Cache for Mobile Deep Visioncited by 57
  31. 2023Distilling Script Knowledge from Large Language Models for Constrained Language Planningcited by 45
  32. 2023LLMZip: Lossless Text Compression using Large Language Modelscited by 44
  33. 2023Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Expertscited by 23
  34. 2023TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Trainingcited by 15
  35. 2022STI: Turbocharge NLP Inference at the Edge via Elastic Pipeliningcited by 14
  36. 2021Enabling Large Neural Networks on Tiny Microcontrollers with Swappingcited by 9
  37. 2017MobiRNN: Efficient Recurrent Neural Network Execution on Mobile GPUcited by 5
  38. 2017CRNN: A Joint Neural Network for Redundancy Detectioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 16; 2025: 18; 2026: 26.

65 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  2. 2023Efficient Large Language Models: A Surveycited by 183
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  4. 2024Small Language Models: Survey, Measurements, and Insightscited by 109
  5. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  7. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
  8. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  9. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  10. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 36
  11. 2023Mixture-of-Linear-Experts for Long-term Time Series Forecastingcited by 29
  12. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inferencecited by 28
38 more
  1. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  2. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  3. 2025Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Surveycited by 13
  4. 2025Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deploymentcited by 10
  5. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  6. 2024The Internet of Things in the Era of Generative AI: Vision and Challengescited by 8
  7. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  8. 2025Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Modelscited by 8
  9. 2026Forget to Improve: On-Device LLM-Agent Continual Learning via Budget-Curated Memorycited by 8
  10. 2024HiRE: High Recall Approximate Top-k Estimation for Efficient LLM Inferencecited by 7
  11. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  12. 2025SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inferencecited by 7
  13. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placementcited by 7
  14. 2025MoE2: Optimizing Collaborative Inference for Edge Large Language Modelscited by 6
  15. 2025The Larger the Merrier? Efficient Large AI Model Inference in Wireless Edge Networkscited by 6
  16. 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inferencecited by 5
  17. 2025Towards Pervasive Distributed Agentic Generative AI -- A State of The Artcited by 4
  18. 2025Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Executioncited by 4
  19. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Schedulingcited by 4
  20. 2026Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneitycited by 4
  21. 2026Agent-X: Full Pipeline Acceleration of On-device AI Agentscited by 4
  22. 2024A General-Purpose Device for Interaction with LLMscited by 3
  23. 2024Scalable Multi-Domain Adaptation of Language Models using Modular Expertscited by 3
  24. 2025CoSMoEs: Compact Sparse Mixture of Expertscited by 3
  25. 2025Every Software as an Agent: Blueprint and Case Studycited by 2
  26. 2025MNN-AECS: Energy Optimization for LLM Decoding on Mobile Devices via Adaptive Core Selectioncited by 2
  27. 2025CoMoE: Collaborative Optimization of Expert Aggregation and Offloading for MoE-based LLMs at Edgecited by 2
  28. 2026DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edgecited by 2
  29. 2026Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robotscited by 2
  30. 2026Parameter-Free Encoders Remain Viable for RDB Foundation Modelscited by 2
  31. 2024All Language Models Large and Smallcited by 1
  32. 2024InterroGate: Learning to Share, Specialize, and Prune Representations for Multi-task Learningcited by 1
  33. 2026Quantization-Aware Collaborative Inference for Large Embodied AI Modelscited by 1
  34. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guaranteescited by 1
  35. 2026Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Studycited by 1
  36. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 0
  37. 2025Resource-Aware LLM Reasoning for Mobile Edge General Intelligencecited by 0
  38. 2026Evaluating Zero-Shot and One-Shot Adaptation of Small Language Models in Leader-Follower Interactioncited by 0

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MobileMoE: Scaling On-Device Mixture of Experts
  2. 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices
  3. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  4. 2026FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices
  5. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving
  6. 2025SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment
  7. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
  8. 2023LinguaLinked: A Distributed Large Language Model Inference System for Mobile Devices