mlsanityData to 7 Oct 2026

Scaling Laws for Upcycling Mixture-of-Experts Language Models

arXiv 2502.03009 ↗First posted 5 Feb 2025cs.LG cs.CLcited by 12

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Pretraining large language models (LLMs) is resource-intensive, often requiring months of training time even with high-end GPU clusters. There are two approaches of mitigating such computational demands: reusing smaller models to train larger ones (upcycling), and training computationally efficient models like mixture-of-experts (MoE). In this paper, we study the upcycling of LLMs to MoE models, of which the scaling behavior remains underexplored. Through extensive experiments, we identify empirical scaling laws that describe how performance depends on dataset size and model configuration. Particularly, we show that, while scaling these factors improves performance, there is a novel interaction term between the dense and upcycled training dataset that limits the efficiency of upcycling at large computational budgets. Based on these findings, we provide guidance to scale upcycling, and establish conditions under which upcycling outperforms from-scratch trainings within budget constraints.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 59 of its 62 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

59 of 62 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent4 Nov
  2. 2024Upcycling Large Language Models into Mixture of Experts10 Oct
  3. 2024GRIN: GRadient-INformed MoE18 Sep
  4. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  5. 2024A Survey on Mixture of Experts in Large Language Models26 Jun
  6. 2024A Closer Look into Mixture-of-Experts in Large Language Models26 Jun
  7. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models3 Jun
  8. 2024Scaling Laws for Fine-Grained Mixture of Experts12 Feb
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  10. 2024Mixtral of Experts8 Jan
  11. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
1 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2024Qwen2 Technical Reportcited by 2,076
34 more
  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2017Decoupled Weight Decay Regularizationcited by 1,722
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  7. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  9. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  11. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  12. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  13. 2023Scaling Data-Constrained Language Modelscited by 391
  14. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  15. 2021Explaining Neural Scaling Lawscited by 353
  16. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  17. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  18. 2021Scaling Laws for Transfercited by 265
  19. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  20. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  21. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  22. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  23. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  24. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  25. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  26. 2022A Solvable Model of Neural Scaling Lawscited by 101
  27. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  28. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  29. 2021Learning Curve Theorycited by 79
  30. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  31. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 66
  32. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  33. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 42
  34. 2019Beyond Human-Level Accuracy: Computational Challenges in Deep Learningcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2025From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetancited by 3
  2. 2025Evaluating the Robustness of Chinchilla Compute-Optimal Scalingcited by 2
  3. 2025Reusing Overtrained Language Models Saturates Scalingcited by 2
  4. 2026Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selectioncited by 2
  5. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Expertscited by 1
  6. 2025Scaling depth capacity via zero/one-layer model expansioncited by 1
  7. 2026Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettercited by 1
  8. 2026Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraintscited by 0
  9. 2026Generalization and Scaling Laws for Mixture-of-Experts Transformerscited by 0
  10. 2026Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcyclingcited by 0
  11. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Expertscited by 0
  12. 2026Toward LLMs Beyond English-Centric Developmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Llama 3 Meets MoE: Efficient Upcycling
  2. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  3. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  4. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
  5. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
  6. 2026Training-Free Dynamic Upcycling of Expert Language Models
  7. 2024Sparse Upcycling: Inference Inefficient Finetuning
  8. 2025Towards a Comprehensive Scaling Law of Mixture-of-Experts