mlsanityData to 14 Oct 2026

Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient

arXiv 2502.05172 ↗First posted 7 Feb 2025cs.LG cs.AI cs.CLcited by 31

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Mixture of Experts (MoE) architectures have significantly increased computational efficiency in both research and real-world applications of large-scale machine learning models. However, their scalability and efficiency under memory constraints remain relatively underexplored. In this work, we present joint scaling laws for dense and MoE models, incorporating key factors such as the number of active parameters, dataset size, and the number of experts. Our findings provide a principled framework for selecting the optimal MoE configuration under fixed memory and compute budgets. Surprisingly, we show that MoE models can be more memory-efficient than dense models, contradicting conventional wisdom. To derive and validate the theoretical predictions of our scaling laws, we conduct over 280 experiments with up to 2.7B active parameters and up to 5B total parameters. These results offer actionable insights for designing and deploying MoE models in practical large-scale training scenarios.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 18 of its 40 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

18 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning11 Sep

Other references · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2020GLU Variants Improve Transformercited by 1,804
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  8. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  9. 2018An Empirical Model of Large-Batch Trainingcited by 281
  10. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  11. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  12. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
5 more
  1. 2024Scaling Laws for Precisioncited by 95
  2. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  3. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 50
  4. 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 39
  5. 2024Toward Inference-optimal Mixture-of-Expert Large Language Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 21.

31 papers.

Sort:

By area:

  1. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 45
  2. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretrainingcited by 34
  3. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 19
  4. 2025Mixture of Experts in Large Language Modelscited by 15
  5. 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 10
  6. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 9
  7. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resourcecited by 8
  8. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  9. 2026Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Modelscited by 4
  10. 2025μ-Parametrization for Mixture of Expertscited by 3
  11. 2026MobileMoE: Scaling On-Device Mixture of Expertscited by 3
  12. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Expertscited by 3
19 more
  1. 2025Evaluating the Robustness of Chinchilla Compute-Optimal Scalingcited by 2
  2. 2025Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimizationcited by 2
  3. 2026Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimizationcited by 2
  4. 2026Slicing and Dicing: Configuring Optimal Mixtures of Expertscited by 2
  5. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Expertscited by 1
  6. 2026Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matchingcited by 1
  7. 2026Toward Calibrated Mixture-of-Experts Under Distribution Shiftcited by 1
  8. 2026LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Modelscited by 1
  9. 2026Scaling Laws for Looped Mixture of Expertscited by 1
  10. 2026CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Modelscited by 0
  11. 2026Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fitscited by 0
  12. 2026Continued AI Scaling Requires Repeated Efficiency Doublingscited by 0
  13. 2026Generalization and Scaling Laws for Mixture-of-Experts Transformerscited by 0
  14. 2026EMO: Frustratingly Easy Progressive Training of Extendable MoEcited by 0
  15. 2026Hyperparameter Scaling Laws Across MoE Sparsitycited by 0
  16. 2026Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivationcited by 0
  17. 2026X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depthscited by 0
  18. 2026Kairos: Toward a Byte-Level Foundation Model for Every Modalitycited by 0
  19. 2026Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
  2. 2024Scaling Laws for Fine-Grained Mixture of Experts
  3. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
  4. 2026Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts
  5. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
  6. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models
  7. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Models
  8. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models