mlsanityData to 7 Oct 2026

MegaBlocks: Efficient Sparse Training with Mixture-of-Experts

arXiv 2211.15841 ↗First posted 29 Nov 2022cs.LG cs.AI cs.DCcited by 204

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

We present MegaBlocks, a system for efficient Mixture-of-Experts (MoE) training on GPUs. Our system is motivated by the limitations of current frameworks, which restrict the dynamic routing in MoE layers to satisfy the constraints of existing software and hardware. These formulations force a tradeoff between model quality and hardware efficiency, as users must choose between dropping tokens from the computation or wasting computation and memory on padding. To address these limitations, we reformulate MoE computation in terms of block-sparse operations and develop new block-sparse GPU kernels that efficiently handle the dynamism present in MoEs. Our approach never drops tokens and maps efficiently to modern hardware, enabling end-to-end training speedups of up to 40% over MoEs trained with the state-of-the-art Tutel library and 2.4x over DNNs trained with the highly-optimized Megatron-LM framework.

Key facts. Cited by 204 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 30 of its 51 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

30 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2017Mixed Precision Trainingcited by 1,135
17 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  2. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  5. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  6. 2019The State of Sparsity in Deep Neural Networkscited by 471
  7. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  8. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  9. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  10. 2021Hash Layers For Large Sparse Modelscited by 236
  11. 2018Efficient Neural Audio Synthesiscited by 218
  12. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  13. 2020Sparse GPU Kernels for Deep Learningcited by 138
  14. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  15. 2019Fast Sparse ConvNetscited by 86
  16. 2017Block-Sparse Recurrent Neural Networkscited by 38
  17. 2018Balanced Sparsity for Efficient DNN Inference on GPUcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 55; 2025: 55; 2026: 82.

204 papers.

Sort:

By area:

  1. 2024Mixtral of Expertscited by 1,785
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  3. 2023Challenges and Applications of Large Language Modelscited by 251
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  5. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  6. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  7. 2023Efficient Large Language Models: A Surveycited by 183
  8. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  9. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  10. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  11. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  12. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
38 more
  1. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  2. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  3. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 61
  4. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 55
  5. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  6. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 53
  7. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scalecited by 48
  8. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  9. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
  10. 2025Scaling Laws for Native Multimodal Modelscited by 47
  11. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  12. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  13. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 38
  14. 2025Training Sparse Mixture Of Experts Text Embedding Modelscited by 32
  15. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
  16. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  17. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 29
  18. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorizationcited by 28
  19. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 28
  20. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
  21. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
  22. 2025Apple Intelligence Foundation Language Models: Tech Report 2025cited by 25
  23. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initializationcited by 23
  24. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
  25. 2024Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlappingcited by 22
  26. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  27. 2025Load Balancing Mixture of Experts with Similarity Preserving Routerscited by 22
  28. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  29. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  30. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 20
  31. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 20
  32. 2025EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Modelscited by 18
  33. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16
  34. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Expertscited by 16
  35. 2024GRIN: GRadient-INformed MoEcited by 16
  36. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
  37. 2024Scattered Mixture-of-Experts Implementationcited by 15
  38. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14

50 most cited of 204 All 204 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core
  2. 2021M6-T: Exploring Sparse Expert Models and Beyond
  3. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System
  4. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts
  5. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  6. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  7. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
  8. 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules