mlsanityData to 14 Oct 2026

Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights

arXiv 2506.02890 ↗First posted 3 Jun 2025cs.LG cs.AI cs.CLcited by 2

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture of Experts (MoE) architectures have emerged as pivotal for scaling Large Language Models (LLMs) efficiently. Fine-grained MoE approaches - utilizing more numerous, smaller experts - have demonstrated potential in improving model convergence and quality. This work proposes a set of training recipes and provides a comprehensive empirical evaluation of fine-grained MoE, directly comparing its scaling properties against standard MoE configurations for models with up to 56B total (17B active) parameters. We investigate convergence speed, model performance on downstream benchmarks, and practical training considerations across various setups. Overall, at the largest scale we show that fine-grained MoE achieves better validation loss and higher accuracy across a set of downstream benchmarks. This study offers empirical grounding and practical insights for leveraging fine-grained MoE in the development of future large-scale models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 36 of its 40 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

36 of 40 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Llama 3 Meets MoE: Efficient Upcycling13 Dec
  2. 2024Sparse Upcycling: Inference Inefficient Finetuning13 Nov
  3. 2024Upcycling Large Language Models into Mixture of Experts10 Oct
  4. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  6. 2024Scaling Laws for Fine-Grained Mixture of Experts12 Feb
  7. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models29 Jan
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  9. 2023LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models25 Sep
  10. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
1 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
11 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  4. 2020GLU Variants Improve Transformercited by 1,804
  5. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  6. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  7. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  8. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  9. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 72
  10. 2024Nemotron-4 15B Technical Reportcited by 36
  11. 2024Scattered Mixture-of-Experts Implementationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulationcited by 3
  2. 2026Generalization and Scaling Laws for Mixture-of-Experts Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
  2. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  3. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
  4. 2024A Survey on Mixture of Experts in Large Language Models
  5. 2025Mixture of Experts in Large Language Models
  6. 2025dots.llm1 Technical Report
  7. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  8. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models