mlsanityData to 7 Oct 2026

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints

arXiv 2212.05055 ↗First posted 9 Dec 2022cs.LG cs.CL cs.CVcited by 207

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Training large, deep neural networks to convergence can be prohibitively expensive. As a result, often only a small selection of popular, dense models are reused across different contexts and tasks. Increasingly, sparsely activated models, which seek to decouple model size from computation costs, are becoming an attractive alternative to dense models. Although more efficient in terms of quality and computation cost, sparse models remain data-hungry and costly to train from scratch in the large scale regime. In this work, we propose sparse upcycling -- a simple way to reuse sunk training costs by initializing a sparsely activated Mixture-of-Experts model from a dense checkpoint. We show that sparsely upcycled T5 Base, Large, and XL language models and Vision Transformer Base and Large models, respectively, significantly outperform their dense counterparts on SuperGLUE and ImageNet, using only ~50% of the initial dense pretraining sunk cost. The upcycled models also outperform sparse models trained from scratch on 100% of the initial dense pretraining computation budget.

Key facts. Cited by 207 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 50 of its 58 references that are in the map. First posted on arXiv 9 Dec 2022.

Builds on

50 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation15 Apr
  2. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
33 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  8. 2021Scaling Vision Transformerscited by 993
  9. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  11. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  12. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  13. 2020What is the State of Neural Network Pruning?cited by 578
  14. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  15. 2019The State of Sparsity in Deep Neural Networkscited by 471
  16. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  17. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  18. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  19. 2021Hash Layers For Large Sparse Modelscited by 236
  20. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  21. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  22. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  23. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  24. 2021The Efficiency Misnomercited by 98
  25. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  26. 2020On the Transformer Growth for Progressive BERT Trainingcited by 49
  27. 2022Staged Training for Transformer Language Modelscited by 49
  28. 2022Residual Mixture of Expertscited by 39
  29. 2020Shallow-to-Deep Training for Neural Machine Translationcited by 36
  30. 2020Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedupcited by 31
  31. 2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31
  32. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gatecited by 30
  33. 2021Speeding up Deep Model Training by Sharing Weights and Then Unsharingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 72; 2025: 60; 2026: 62.

207 papers.

Sort:

By area:

  1. 2024Qwen2 Technical Reportcited by 2,076
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  4. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  6. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  7. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  8. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  9. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  10. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  11. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  12. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
38 more
  1. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  2. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  3. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  4. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  5. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  6. 2024Stable LM 2 1.6B Technical Reportcited by 86
  7. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  8. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  9. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 61
  10. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  11. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  12. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Modelscited by 57
  13. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  14. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  15. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 53
  16. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  17. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  18. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  19. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  20. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 44
  21. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  22. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  23. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  24. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Modelscited by 36
  25. 2025Training Sparse Mixture Of Experts Text Embedding Modelscited by 32
  26. 2023Robust Mixture-of-Expert Training for Convolutional Neural Networkscited by 30
  27. 2024Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large Language Modelscited by 25
  28. 2025Apple Intelligence Foundation Language Models: Tech Report 2025cited by 25
  29. 2025Multilingual Routing in Mixture-of-Expertscited by 25
  30. 2024EVLM: An Efficient Vision-Language Model for Visual Understandingcited by 23
  31. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  32. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initializationcited by 23
  33. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Taskscited by 22
  34. 2024Routers in Vision Mixture of Experts: An Empirical Studycited by 22
  35. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 22
  36. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  37. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 22
  38. 2025Mixture of Experts Made Intrinsically Interpretablecited by 22

50 most cited of 207 All 207 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling
  2. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  3. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  4. 2024Llama 3 Meets MoE: Efficient Upcycling
  5. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  6. 2023Accurate Neural Network Pruning Requires Rethinking Sparse Optimization
  7. 2025MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
  8. 2024Sparse Checkpointing for Fast and Reliable MoE Training