mlsanityData to 7 Oct 2026

Optimizing Mixture of Experts using Dynamic Recompilations

arXiv 2205.01848 ↗First posted 4 May 2022cs.LG cs.DCcited by 4

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

The Mixture of Experts architecture allows for outrageously large neural networks by scaling model parameter size independently from computational demand (FLOPs). However, current DNN frameworks cannot effectively support the dynamic data flow in Mixture of Experts, and implementations on top of these frameworks need to use workarounds that introduce significant overheads. To address the limitation of these frameworks, we present DynaMoE, a DNN library that uses dynamic recompilations to optimize and adapt the use of computational resources to the dynamic needs of Mixture of Experts models. Our evaluation shows that DynaMoE achieves a 1.8x speedup and supports 2.3x larger model sizes when compared to existing MoE systems, even when not using recompilations. We then present further optimizations enabled by dynamic recompilations that yield an additional 1.7x speedup while simultaneously reducing memory pressure and improving model quality.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 15 of its 31 references that are in the map. First posted on arXiv 4 May 2022.

Builds on

15 of 31 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  8. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  9. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
  10. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  11. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  12. 2018Tensor2Tensor for Neural Machine Translationcited by 85
3 more
  1. 2016Theano: A Python framework for fast computation of mathematical expressionscited by 64
  2. 2019Chainer: A Deep Learning Framework for Accelerating the Research Cyclecited by 22
  3. 2017DyNet: The Dynamic Neural Network Toolkitcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 2; 2025: 0; 2026: 1.

4 papers.

  1. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  2. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  3. 2024Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMscited by 4
  4. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks
  2. 2026Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
  3. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
  4. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
  5. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  6. 2026Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts
  7. 2025Mixture of Raytraced Experts
  8. 2026Structure Aware Neural Architecture Search for Mixture of Experts