mlsanityData to 7 Oct 2026

Sparse Upcycling: Inference Inefficient Finetuning

arXiv 2411.08968 ↗First posted 13 Nov 2024cs.LG cs.CLcited by 1

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Small, highly trained, open-source large language models are widely used due to their inference efficiency, but further improving their quality remains a challenge. Sparse upcycling is a promising approach that transforms a pretrained dense model into a Mixture-of-Experts (MoE) architecture, increasing the model's parameter count and quality. In this work, we compare the effectiveness of sparse upcycling against continued pretraining (CPT) across different model sizes, compute budgets, and pretraining durations. Our experiments show that sparse upcycling can achieve better quality, with improvements of over 20% relative to CPT in certain scenarios. However, this comes with a significant inference cost, leading to 40% slowdowns in high-demand inference settings for larger models. Our findings highlight the trade-off between model quality and inference efficiency, offering insights for practitioners seeking to balance model quality and deployment constraints.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 30 of its 38 references that are in the map. First posted on arXiv 13 Nov 2024.

Builds on

30 of 38 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts15 Aug
  2. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models3 Jun
  3. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM12 Mar
  4. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 25

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
13 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  3. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  6. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  7. 2018CoQA: A Conversational Question Answering Challengecited by 674
  8. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  9. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  10. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  11. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  12. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  13. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 51

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insightscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Upcycling Large Language Models into Mixture of Experts
  2. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Models
  3. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
  4. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  5. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
  6. 2026SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling
  7. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
  8. 2024Llama 3 Meets MoE: Efficient Upcycling