mlsanityData to 7 Oct 2026

Scattered Mixture-of-Experts Implementation

arXiv 2403.08245 ↗First posted 13 Mar 2024cs.LG cs.DCcited by 15

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

We present ScatterMoE, an implementation of Sparse Mixture-of-Experts (SMoE) on GPUs. ScatterMoE builds upon existing implementations, and overcoming some of the limitations to improve inference and training speed, and memory footprint. This implementation achieves this by avoiding padding and making excessive copies of the input. We introduce ParallelLinear, the main component we use to build our implementation and the various kernels used to speed up the operation. We benchmark our implementation against Megablocks, and show that it enables a higher throughput and lower memory footprint. We also show how ParallelLinear enables extension of the Mixture-of-Experts concept by demonstrating with an implementation of Mixture of Attention.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 15 of its 20 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

15 of 20 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov

Other references · 14

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2024Mixtral of Expertscited by 1,785
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  8. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  9. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  10. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 58
  11. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attentioncited by 34
  12. 2023Sparse Universal Transformercited by 28
2 more
  1. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
  2. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 2; 2026: 6.

15 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  2. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  3. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 41
  4. 2024Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Schedulercited by 26
  5. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 20
  6. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 20
  7. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inferencecited by 6
  8. 2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
  9. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insightscited by 2
  10. 2026Grouped Query Experts: Mixture-of-Experts on GQA Self-Attentioncited by 2
  11. 2026OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scalecited by 0
  12. 2026RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Expertscited by 0
3 more
  1. 2026Mixture-of-Kittens: MoE Megakernel for NVL72scited by 0
  2. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routingcited by 0
  3. 2026Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts
  2. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  3. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  4. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
  5. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
  6. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models
  7. 2025HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs
  8. 2026Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference