mlsanityData to 13 Oct 2026

SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models

arXiv 2310.18859 ↗First posted 29 Oct 2023cs.LG cs.DCcited by 38

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) has emerged as a favorable architecture in the era of large models due to its inherent advantage, i.e., enlarging model capacity without incurring notable computational overhead. Yet, the realization of such benefits often results in ineffective GPU memory utilization, as large portions of the model parameters remain dormant during inference. Moreover, the memory demands of large models consistently outpace the memory capacity of contemporary GPUs. Addressing this, we introduce SiDA-MoE (Sparsity-inspired Data-Aware), an efficient inference approach tailored for large MoE models. SiDA-MoE judiciously exploits both the system's main memory, which is now abundant and readily scalable, and GPU memory by capitalizing on the inherent sparsity on expert activation in MoE models. By adopting a data-aware perspective, SiDA-MoE achieves enhanced model efficiency with a neglectable performance drop. Specifically, SiDA-MoE attains a remarkable speedup in MoE inference with up to 3.93× throughput increasing, up to 72% latency reduction, and up to 80% GPU memory saving with down to 1% performance drop. This work paves the way for scalable and efficient deployment of large MoE models, even with constrained resources. Code is available at: https://github.com/timlee0212/SiDA-MoE.

Key facts. Cited by 38 papers in the map (data through 13 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 64 of its 83 references that are in the map. First posted on arXiv 29 Oct 2023.

Builds on

64 of 83 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices28 Aug
  2. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  3. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May
  4. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  5. 2023Segment Anythingcited by 8,195
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
  7. 2020Scaling Laws for Neural Language Modelscited by 6,494
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,339
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,539
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
38 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,215
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,884
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,725
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,618
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,365
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,272
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,103
  8. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 787
  9. 2021Scaling Vision with Sparse Mixture of Expertscited by 744
  10. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 646
  11. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  12. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 413
  13. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 406
  14. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 406
  15. 2023Adapting Language Models to Compress Contextscited by 351
  16. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 339
  17. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 322
  18. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  19. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  20. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 276
  21. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 252
  22. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 250
  23. 2021Hash Layers For Large Sparse Modelscited by 237
  24. 2019Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computingcited by 226
  25. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 194
  26. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 161
  27. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  28. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  29. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 133
  30. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  31. 2019Convergence of Edge Computing and Deep Learning: A Comprehensive Surveycited by 120
  32. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 115
  33. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 104
  34. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
  35. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  36. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  37. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  38. 2023PB-LLM: Partially Binarized Large Language Modelscited by 81

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 6; 2025: 15; 2026: 17.

38 papers.

Sort:

By area:

  1. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  3. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
  4. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  5. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  6. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipelinecited by 21
  7. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14
  8. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inferencecited by 14
  9. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gatecited by 13
  10. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inferencecited by 12
  11. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  12. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
26 more
  1. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanismcited by 9
  2. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  3. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 8
  4. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placementcited by 7
  5. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  6. 2025From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routingcited by 5
  7. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  8. 2025MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancingcited by 4
  9. 2025DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurancecited by 4
  10. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Schedulingcited by 4
  11. 2026Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneitycited by 4
  12. 2025CoMoE: Collaborative Optimization of Expert Aggregation and Offloading for MoE-based LLMs at Edgecited by 2
  13. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCscited by 2
  14. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardwarecited by 2
  15. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Expertscited by 1
  16. 2025Efficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborativecited by 0
  17. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0
  18. 2026Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Expertscited by 0
  19. 2026Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placementcited by 0
  20. 2026HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inferencecited by 0
  21. 2026MoE Expert Execution in Disaggregated LLM Serving with a High-Bandwidth ReRAM Near-Memory Architecturecited by 0
  22. 2026Budgeting Bytes: A Windowed Storage Roofline and Dual-Budget Architecture Ablations for Storage-Bound LLM Decodingcited by 0
  23. 2026A Multi-Engine Dataflow for MoE Decoding on Scratchpad-Based Tensor Acceleratorscited by 0
  24. 2026TopoCompress: Topology Aware Token Compression Algorithm for Distributed Edge MoE Inferencecited by 0
  25. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Stagingcited by 0
  26. 2026What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Computecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
  2. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
  3. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  4. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  5. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  6. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  7. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  8. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts