mlsanityData to 14 Oct 2026

SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget

arXiv 2308.15030 ↗First posted 29 Aug 2023cs.AIcited by 27

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture of experts (MoE) is a popular technique to improve capacity of Large Language Models (LLMs) with conditionally-activated parallel experts. However, serving MoE models on memory-constrained devices is challenging due to the large parameter size. Typical solutions such as memory swapping or expert pruning may lead to significantly higher latency or severe accuracy loss. In this paper, we introduce SwapMoE, a framework for efficient serving of MoE-based large language models with tunable memory budgets. The main idea of SwapMoE is to keep a small dynamic set of important experts, namely Virtual Experts, in the main memory for inference, while seamlessly maintaining how the Virtual Experts map to the actual experts. Experiments have shown that SwapMoE can reduce the memory footprint while maintaining reasonable accuracy. For example, on text summarization tasks with Switch Transformer, SwapMoE can reduce the memory consumption from 14.2 GiB to 4.7 GiB, together with 50% latency reduction and a slight Rouge-2 score drop of 0.041.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 19 of its 36 references that are in the map. First posted on arXiv 29 Aug 2023.

Builds on

19 of 36 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference10 Mar
  2. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  3. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
  4. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 13

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  5. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  6. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  7. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  8. 2018NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Visioncited by 68
  9. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37
  10. 2021LegoDNN: Block-grained Scaling of Deep Neural Networks for Mobile Visioncited by 20
  11. 2023AdaptiveNet: Post-deployment Neural Architecture Adaptation for Diverse Edge Environmentscited by 19
  12. 2020Dynamic Slicing for Deep Neural Networkscited by 16
1 more
  1. 2023Cost-effective On-device Continual Learning over Memory Hierarchy with Mirocited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 15; 2026: 8.

27 papers.

Sort:

By area:

  1. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  2. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Cachingcited by 40
  3. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  4. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  5. 2026DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inferencecited by 24
  6. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14
  7. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inferencecited by 13
  8. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restorationcited by 12
  9. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  10. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placementcited by 7
  11. 2025Dynamic Expert Quantization for Scalable Mixture-of-Experts Inferencecited by 7
  12. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5
15 more
  1. 2025DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurancecited by 5
  2. 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inferencecited by 5
  3. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Schedulingcited by 4
  4. 2025Efficient Deployment of Vision-Language Models on Mobile Devices: A Case Study on OnePlus 13Rcited by 2
  5. 2025CoMoE: Collaborative Optimization of Expert Aggregation and Offloading for MoE-based LLMs at Edgecited by 2
  6. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  7. 2026MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Servingcited by 2
  8. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 1
  9. 2025MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?cited by 1
  10. 2025Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systemscited by 1
  11. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMscited by 0
  12. 2026Networking-Aware Energy Efficiency in Agentic AI Inference: A Surveycited by 0
  13. 2026VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloadingcited by 0
  14. 2026Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inferencecited by 0
  15. 2026MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  2. 2025Mixture of Experts in Large Language Models
  3. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  4. 2026LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing
  5. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference
  6. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
  7. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices
  8. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models