# Scattered Mixture-of-Experts Implementation

arXiv: https://arxiv.org/abs/2403.08245 · HTML: https://mlsanity.com/paper/2403.08245/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 15 of its 20 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 15 of 15)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Reducing Activation Recomputation in Large Transformer Models](https://mlsanity.com/paper/2205.05198/) (2022)
- [MegaBlocks: Efficient Sparse Training with Mixture-of-Experts](https://mlsanity.com/paper/2211.15841/) (2022)
- [Scaling Laws for Fine-Grained Mixture of Experts](https://mlsanity.com/paper/2402.07871/) (2024)
- [Mixture of Attention Heads: Selecting Attention Heads Per Token](https://mlsanity.com/paper/2210.05144/) (2022)
- [SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention](https://mlsanity.com/paper/2312.07987/) (2023)
- [Sparse Universal Transformer](https://mlsanity.com/paper/2310.07096/) (2023)
- [Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production](https://mlsanity.com/paper/2211.10017/) (2022)
- [PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformation](https://mlsanity.com/paper/2301.10936/) (2023)

## Built on by (most cited first, 15 of 15)

- [A Survey on Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2407.06204/) (2024)
- [ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing](https://mlsanity.com/paper/2412.14711/) (2024)
- [Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models](https://mlsanity.com/paper/2404.05567/) (2024)
- [Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler](https://mlsanity.com/paper/2408.13359/) (2024)
- [Mixture of Parrots: Experts improve memorization more than reasoning](https://mlsanity.com/paper/2410.19034/) (2024)
- [SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations](https://mlsanity.com/paper/2512.14080/) (2025)
- [EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference](https://mlsanity.com/paper/2410.12247/) (2024)
- [Exploring the Benefit of Activation Sparsity in Pre-training](https://mlsanity.com/paper/2410.03440/) (2024)
- [Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights](https://mlsanity.com/paper/2506.02890/) (2025)
- [Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention](https://mlsanity.com/paper/2606.20945/) (2026)
- [OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale](https://mlsanity.com/paper/2602.05711/) (2026)
- [RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts](https://mlsanity.com/paper/2604.26039/) (2026)
- [Mixture-of-Kittens: MoE Megakernel for NVL72s](https://mlsanity.com/paper/2609.36070/) (2026)
- [MoRE: Scaling mixture of experts with hardware-aware low-rank routing](https://mlsanity.com/paper/2609.36301/) (2026)
- [Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization](https://mlsanity.com/paper/2610.00558/) (2026)

## Abstract (arXiv)

We present ScatterMoE, an implementation of Sparse Mixture-of-Experts (SMoE) on GPUs. ScatterMoE builds upon existing implementations, and overcoming some of the limitations to improve inference and training speed, and memory footprint. This implementation achieves this by avoiding padding and making excessive copies of the input. We introduce ParallelLinear, the main component we use to build our implementation and the various kernels used to speed up the operation. We benchmark our implementation against Megablocks, and show that it enables a higher throughput and lower memory footprint. We also show how ParallelLinear enables extension of the Mixture-of-Experts concept by demonstrating with an implementation of Mixture of Attention.