mlsanityData to 14 Oct 2026

Scaling Laws for Fine-Grained Mixture of Experts

arXiv 2402.07871 ↗First posted 12 Feb 2024cs.LG cs.AI cs.CLcited by 173

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, incorporating an expanded range of variables. Specifically, we introduce a new hyperparameter, granularity, whose adjustment enables precise control over the size of the experts. Building on this, we establish scaling laws for fine-grained MoE, taking into account the number of training tokens, model size, and granularity. Leveraging these laws, we derive the optimal training configuration for a given computational budget. Our findings not only show that MoE models consistently outperform dense Transformers but also highlight that the efficiency gap between dense and MoE models widens as we scale up the model size and training budget. Furthermore, we demonstrate that the common practice of setting the size of experts in MoE to mirror the feed-forward layer is not optimal at almost any computational budget.

Key facts. Cited by 173 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 13 of its 25 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

13 of 25 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model23 May
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 10

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  6. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  7. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  8. 2021Hash Layers For Large Sparse Modelscited by 237
  9. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  10. 2021Scaling Laws for Neural Machine Translationcited by 111

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 32; 2025: 63; 2026: 78.

173 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  2. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 138
  3. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  4. 2024Scaling Laws for Precisioncited by 95
  5. 2024Mixture of A Million Expertscited by 72
  6. 2024MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Expertscited by 70
  7. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 59
  8. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 54
  9. 2025Scaling Laws for Optimal Data Mixturescited by 52
  10. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 50
  11. 2025Scaling Laws for Native Multimodal Modelscited by 50
  12. 2024Upcycling Large Language Models into Mixture of Expertscited by 49
38 more
  1. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  2. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 45
  3. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
  4. 2024Scaling Optimal LR Across Token Horizonscited by 40
  5. 2025Generative Frame Sampler for Long Video Understandingcited by 37
  6. 2025Training Sparse Mixture Of Experts Text Embedding Modelscited by 32
  7. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
  8. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  9. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 27
  10. 2024Scaling Laws for Multilingual Language Modelscited by 23
  11. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  12. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  13. 2024Scaling Law for Language Models Training Considering Batch Sizecited by 22
  14. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  15. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 21
  16. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 21
  17. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 20
  18. 2024Ultra-Sparse Memory Networkcited by 19
  19. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 19
  20. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 18
  21. 2025EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Modelscited by 18
  22. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 18
  23. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Corecited by 17
  24. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Expertscited by 16
  25. 2025Scaling Inference-Efficient Language Modelscited by 16
  26. 2024Scattered Mixture-of-Experts Implementationcited by 15
  27. 2024Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scalecited by 15
  28. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Modelscited by 15
  29. 2025Value-Based Deep RL Scales Predictablycited by 15
  30. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14
  31. 2024Monet: Mixture of Monosemantic Experts for Transformerscited by 13
  32. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 13
  33. 2025Can Language Models Discover Scaling Laws?cited by 13
  34. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 13
  35. 2026Prescriptive Scaling Laws for Data Constrained Trainingcited by 12
  36. 2024Predicting Emergent Capabilities by Finetuningcited by 10
  37. 2025Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptationcited by 10
  38. 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 10

50 most cited of 173 All 173 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
  2. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  3. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  4. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  5. 2025Advancing Expert Specialization for Better MoE
  6. 2025Mixture of Experts in Large Language Models
  7. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  8. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource