mlsanityData to 14 Oct 2026

Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts

arXiv 2510.08008 ↗First posted 9 Oct 2025cs.LGcited by 1

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

As the computational demands for pre-training Large Language Models (LLMs) continue to surge, the need for efficient training paradigms becomes critical. Despite the vast resources already invested in existing pre-trained checkpoints, these assets often remain under-leveraged due to architectural limitations. We introduce an "orthogonal growth" strategy designed to "recycle" these checkpoints by strategically expanding their parameters prior to continued training. Our method focuses on optimizing converged Mixture-of-Experts (MoE) models through two dimensions: interpositional layer copying for increased depth and noisy expert duplication for expanded width. Through extensive scaling laws analysis, we demonstrate a strong positive correlation between the "sunk cost" (prior investment) and the final model accuracy. Empirical results on models up to 70B parameters and 1T tokens show that our recycling approach yields a 10.6% accuracy improvement compared to training from scratch under identical extra compute budgets. This work provides a cost-effective blueprint for sustainable large-scale LLM development.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 48 of its 55 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

48 of 55 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts11 Aug
  2. 2025Qwen3 Technical Report14 May
  3. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization26 Feb
  4. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Models5 Feb
  5. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent4 Nov
  6. 2024Upcycling Large Language Models into Mixture of Experts10 Oct
  7. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  8. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models3 Jun
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  10. 2024Mixtral of Experts8 Jan
  11. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 36

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2024Qwen2 Technical Reportcited by 2,120
  11. 2020Knowledge Distillation: A Surveycited by 1,603
  12. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
24 more
  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  2. 2017Mixed Precision Trainingcited by 1,156
  3. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  5. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  6. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  7. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  8. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  9. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  10. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  11. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  12. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  13. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  14. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  15. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 83
  16. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  17. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  18. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 66
  19. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  20. 2022Staged Training for Transformer Language Modelscited by 49
  21. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 43
  22. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 33
  23. 2023FLM-101B: An Open LLM and How to Train It with 100K Budgetcited by 22
  24. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponentscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  2. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  3. 2025Advancing Expert Specialization for Better MoE
  4. 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts
  5. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  6. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
  7. 2026LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
  8. 2025Model Merging in Pre-training of Large Language Models