SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling
On the map: Mixture-of-experts → MoE large language models
Abstract (arXiv)
Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive. MoE upcycling mitigates this cost by converting pretrained dense models into sparse MoE models. However, existing upcycling methods typically rely on large-scale continued training and often perform poorly under data-constrained supervised adaptation, due to either homogeneous experts or overly disruptive perturbations to pretrained parameters. In this setting, effective upcycling must leverage pretrained weight structure while introducing sufficient diversity among routed experts. To this end, we propose SVD-Partitioned Residual Initialization (SPRI), which distributes SVD-partitioned residuals derived from pretrained feed-forward network (FFN) weights across routed experts, introducing controlled expert diversity grounded in pretrained spectral structure. We further introduce a two-stage training strategy to improve adaptation stability. We evaluate SPRI on multilingual speech-to-text translation, where limited supervised data challenges MoE upcycling and multiple target languages provide natural routing heterogeneity. On CoVoST2 across 15 En-to-XX directions, SPRI improves average BLEU and COMET over fully fine-tuned dense models by 2.58 and 3.32 points, respectively, and outperforms the prior best MoE upcycling baseline by 3.39 BLEU and 4.34 COMET points.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 16 of its 22 references that are in the map. First posted on arXiv 15 Jun 2026.
Builds on
16 of 22 references.
Closest ancestors · 9
Same direction, newest first
- 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization26 Feb
- 2024Upcycling Large Language Models into Mixture of Experts10 Oct
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
- 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM12 Mar
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
- 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
- 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan
Other references · 7
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
- 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
- 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 74
- 2018Universal Neural Machine Translation for Extremely Low Resource Languagescited by 66
- 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Sparse Upcycling: Inference Inefficient Finetuning
- 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference
- 2025Scaling Laws for Upcycling Mixture-of-Experts Language Models
- 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
- 2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging
- 2026DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition
- 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
- 2026Training-Free Dynamic Upcycling of Expert Language Models