Llama 3 Meets MoE: Efficient Upcycling
On the map: Mixture-of-experts → MoE large language models
Abstract (arXiv)
Scaling large language models (LLMs) significantly improves performance but comes with prohibitive computational costs. Mixture-of-Experts (MoE) models offer an efficient alternative, increasing capacity without a proportional rise in compute requirements. However, training MoE models from scratch poses challenges like overfitting and routing instability. We present an efficient training recipe leveraging pre-trained dense checkpoints, training an 8-Expert Top-2 MoE model from Llama 3-8B with less than 1% of typical pre-training compute. Our approach enhances downstream performance on academic benchmarks, achieving a 2% improvement in 0-shot accuracy on MMLU, while reaching a Model FLOPs Utilization (MFU) of 46.8% during training using our framework. We also integrate online upcycling in NeMo for seamless use of pre-trained weights, enabling cost-effective development of high-capacity MoE models.
Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 6 of its 34 references that are in the map. First posted on arXiv 13 Dec 2024.
Builds on
6 of 34 references.
Closest ancestors · 1
Same direction, newest first
Other references · 5
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
- 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
Built on by
Citing papers per year: 2025: 4; 2026: 2.
6 papers.
- 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
- 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 19
- 2025The Race to Efficiency: A New Perspective on AI Scaling Lawscited by 3
- 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insightscited by 2
- 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 1
- 2026FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approachcited by 0
- 2026FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach9 Mar 2026
- 2026Scalable Training of Mixture-of-Experts Models with Megatron Core8 Mar 2026
- 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices26 Aug 2025
- 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights3 Jun 2025
- 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications10 Mar 2025
- 2025The Race to Efficiency: A New Perspective on AI Scaling Laws4 Jan 2025
Similar, not cited
Close in wording, no citation either way
- 2024Upcycling Large Language Models into Mixture of Experts
- 2025Scaling Laws for Upcycling Mixture-of-Experts Language Models
- 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
- 2026DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition
- 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
- 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
- 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
- 2026Training-Free Dynamic Upcycling of Expert Language Models