mlsanityData to 14 Oct 2026

SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution

arXiv 2508.18983 ↗First posted 26 Aug 2025cs.AIcited by 3

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The Mixture of Experts (MoE) architecture has emerged as a key technique for scaling Large Language Models by activating only a subset of experts per query. Deploying MoE on consumer-grade edge hardware, however, is constrained by limited device memory, making dynamic expert offloading essential. Unlike prior work that treats offloading purely as a scheduling problem, we leverage expert importance to guide decisions, substituting low-importance activated experts with functionally similar ones already cached in GPU memory, thereby preserving accuracy. As a result, this design reduces memory usage and data transfer, while largely eliminating PCIe overhead. In addition, we introduce a scheduling policy that maximizes the reuse ratio of GPU-cached experts, further boosting efficiency. Extensive evaluations show that our approach delivers 48% lower decoding latency with over 60% expert cache hit rate, while maintaining nearly lossless accuracy.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 32 of its 54 references that are in the map. First posted on arXiv 26 Aug 2025.

Builds on

32 of 54 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025EC2MoE: Adaptive End-Cloud Pipeline Collaboration Enabling Scalable Mixture-of-Experts Inference8 Aug
  2. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving17 Apr
  3. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference8 Apr
  4. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference16 Dec
  5. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  6. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct

Other references · 26

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  9. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  11. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  12. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
14 more
  1. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  2. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  3. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  4. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  5. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  6. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  7. 2022Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Expertscited by 50
  8. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  9. 2023Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Expertscited by 23
  10. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18
  11. 2024Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLMcited by 18
  12. 2025The MoE-Empowered Edge LLMs Deployment: Architecture, Challenges, and Opportunitiescited by 5
  13. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Modelscited by 5
  14. 2024Region-based Content Enhancement for Efficient Video Analytics at the Edgecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inferencecited by 3
  2. 2026FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Executioncited by 3
  3. 2026SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PCcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  2. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
  3. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  4. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
  5. 2026FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
  6. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
  7. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  8. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference