mlsanityData to 14 Oct 2026

Upcycling Large Language Models into Mixture of Experts

arXiv 2410.07524 ↗First posted 10 Oct 2024cs.CL cs.AI cs.LGcited by 49

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Upcycling pre-trained dense language models into sparse mixture-of-experts (MoE) models is an efficient approach to increase the model capacity of already trained models. However, optimal techniques for upcycling at scale remain unclear. In this work, we conduct an extensive study of upcycling methods and hyperparameters for billion-parameter scale language models. We propose a novel "virtual group" initialization scheme and weight scaling approach to enable upcycling into fine-grained MoE architectures. Through ablations, we find that upcycling outperforms continued dense model training. In addition, we show that softmax-then-topK expert routing improves over topK-then-softmax approach and higher granularity MoEs can help improve accuracy. Finally, we upcycled Nemotron-4 15B on 1T tokens and compared it to a continuously trained version of the same model on the same 1T tokens: the continuous trained model achieved 65.3% MMLU, whereas the upcycled model achieved 67.6%. Our results offer insights and best practices to effectively leverage upcycling for building MoE language models. Code is available.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 39 of its 41 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

39 of 41 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  2. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts15 Aug
  3. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models3 Jun
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  5. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM12 Mar
  6. 2024Scaling Laws for Fine-Grained Mixture of Experts12 Feb
  7. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models29 Jan
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  9. 2024Mixtral of Experts8 Jan
  10. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  11. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
2 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2024Qwen2 Technical Reportcited by 2,120
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2020GLU Variants Improve Transformercited by 1,804
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  9. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  11. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  12. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
13 more
  1. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  2. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  3. 2024Nemotron-4 340B Technical Reportcited by 144
  4. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  5. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 83
  6. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 72
  7. 2024Nemotron-4 15B Technical Reportcited by 36
  8. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  9. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 24
  10. 2024Towards an empirical understanding of MoE design choicescited by 20
  11. 2023Reusing Pretrained Models by Multi-linear Operators for Efficient Trainingcited by 15
  12. 2023Composable Function-preserving Expansions for Transformer Architecturescited by 13
  13. 2024Efficient Stagewise Pretraining via Progressive Subnetworkscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 28; 2026: 18.

49 papers.

Sort:

By area:

  1. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 55
  2. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Trainingcited by 26
  3. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initializationcited by 24
  4. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignmentcited by 23
  5. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 19
  6. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Corecited by 17
  7. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 13
  8. 2025DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Modelscited by 10
  9. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resourcecited by 8
  10. 2025FreqMoE: Dynamic Frequency Enhancement for Neural PDE Solverscited by 7
  11. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  12. 2025MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptationcited by 5
37 more
  1. 2025SpikingBrain: Spiking Brain-inspired Large Modelscited by 5
  2. 2025X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compressioncited by 4
  3. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognitioncited by 4
  4. 2025From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetancited by 4
  5. 2025Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcyclingcited by 4
  6. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  7. 2024Sparse Checkpointing for Fast and Reliable MoE Trainingcited by 3
  8. 2025The Race to Efficiency: A New Perspective on AI Scaling Lawscited by 3
  9. 2025Mix-of-Language-Experts Architecture for Multilingual Programmingcited by 3
  10. 2025AWM: Accurate Weight-Matrix Fingerprint for Large Language Modelscited by 3
  11. 2026Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantizationcited by 3
  12. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcyclingcited by 3
  13. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insightscited by 2
  14. 2025Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEscited by 2
  15. 2025Matrix-Driven Identification and Reconstruction of LLM Weight Homologycited by 1
  16. 2025Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcyclingcited by 1
  17. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Expertscited by 1
  18. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Expertscited by 1
  19. 2025Accelerating Frontier MoE Training with 3D Integrated Opticscited by 1
  20. 2025Scaling depth capacity via zero/one-layer model expansioncited by 1
  21. 2026Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettercited by 1
  22. 2026Sparse Spectral LoRA: Routed Experts for Medical VLMscited by 1
  23. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Expertscited by 1
  24. 2025Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Expertscited by 0
  25. 2025UpSafe^circC: Upcycling for Controllable Safety in Large Language Modelscited by 0
  26. 2026ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patternscited by 0
  27. 2026FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approachcited by 0
  28. 2026EMO: Frustratingly Easy Progressive Training of Extendable MoEcited by 0
  29. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Surveycited by 0
  30. 2026Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scalingcited by 0
  31. 2026SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcyclingcited by 0
  32. 2026UMoE:Unlocking Every Expert in Domain-Specific Trainingcited by 0
  33. 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognitioncited by 0
  34. 2026Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillationcited by 0
  35. 2026KITE: KV-Invariant Transformer Expansion for Efficient Agentic LLM Scalingcited by 0
  36. 2026DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Compositioncited by 0
  37. 2026STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modulescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Llama 3 Meets MoE: Efficient Upcycling
  2. 2024Sparse Upcycling: Inference Inefficient Finetuning
  3. 2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging
  4. 2026Training-Free Dynamic Upcycling of Expert Language Models
  5. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  6. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  7. 2024MoIN: Mixture of Introvert Experts to Upcycle an LLM
  8. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router