# Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization

arXiv: https://arxiv.org/abs/2502.19261 · HTML: https://mlsanity.com/paper/2502.19261/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 39 of its 48 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)

## Built on by (most cited first, 24 of 24)

- [Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts](https://mlsanity.com/paper/2508.07785/) (2025)
- [MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation](https://mlsanity.com/paper/2506.14436/) (2025)
- [Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoE](https://mlsanity.com/paper/2605.02641/) (2026)
- [Solar Open 2 Technical Report](https://mlsanity.com/paper/2607.20062/) (2026)
- [Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs](https://mlsanity.com/paper/2510.01185/) (2025)
- [Mixture of Group Experts for Learning Invariant Representations](https://mlsanity.com/paper/2504.09265/) (2025)
- [Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts](https://mlsanity.com/paper/2509.18542/) (2025)
- [Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts](https://mlsanity.com/paper/2510.08008/) (2025)
- [Scaling depth capacity via zero/one-layer model expansion](https://mlsanity.com/paper/2511.04981/) (2025)
- [EngGPT2: Sovereign, Efficient and Open Intelligence](https://mlsanity.com/paper/2603.16430/) (2026)
- [Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts](https://mlsanity.com/paper/2604.19835/) (2026)
- [UniPool: A Globally Shared Expert Pool for Mixture-of-Experts](https://mlsanity.com/paper/2605.06665/) (2026)
- [ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns](https://mlsanity.com/paper/2602.15521/) (2026)
- [Performance Evaluation of Open-Source Large Language Models for Assisting Pathology Report Writing in Japanese](https://mlsanity.com/paper/2603.11597/) (2026)
- [FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach](https://mlsanity.com/paper/2603.13364/) (2026)
- [Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling](https://mlsanity.com/paper/2604.13508/) (2026)
- [EMO: Frustratingly Easy Progressive Training of Extendable MoE](https://mlsanity.com/paper/2605.13247/) (2026)
- [Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling](https://mlsanity.com/paper/2606.07404/) (2026)
- [SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling](https://mlsanity.com/paper/2606.16456/) (2026)
- [Fleet: Few Shots Lead Effective AI-generated Image Detection](https://mlsanity.com/paper/2606.31082/) (2026)
- [UMoE:Unlocking Every Expert in Domain-Specific Training](https://mlsanity.com/paper/2607.11444/) (2026)
- [A Persistent State for Auditable Mixture-of-Experts Routing](https://mlsanity.com/paper/2609.34634/) (2026)
- [DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition](https://mlsanity.com/paper/2610.11317/) (2026)
- [STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules](https://mlsanity.com/paper/2610.12599/) (2026)

## Abstract (arXiv)

The Mixture of Experts (MoE) architecture reduces the training and inference cost significantly compared to a dense model of equivalent capacity. Upcycling is an approach that initializes and trains an MoE model using a pre-trained dense model. While upcycling leads to initial performance gains, the training progresses slower than when trained from scratch, leading to suboptimal performance in the long term. We propose Drop-Upcycling - a method that effectively addresses this problem. Drop-Upcycling combines two seemingly contradictory approaches: utilizing the knowledge of pre-trained dense models while statistically re-initializing some parts of the weights. This approach strategically promotes expert specialization, significantly enhancing the MoE model's efficiency in knowledge acquisition. Extensive large-scale experiments demonstrate that Drop-Upcycling significantly outperforms previous MoE construction methods in the long term, specifically when training on hundreds of billions of tokens or more. As a result, our MoE model with 5.9B active parameters achieves comparable performance to a 13B dense model in the same model family, while requiring approximately 1/4 of the training FLOPs. All experimental resources, including source code, training data, model checkpoints and logs, are publicly available to promote reproducibility and future research on MoE.