mlsanityData to 14 Oct 2026

Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts

arXiv 2506.12597 ↗First posted 14 Jun 2025cs.LGcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We present Sparse Interpolated Mixture-of-Experts (SIMoE) instruction-tuning, an end-to-end algorithm designed to fine-tune a dense pre-trained Large Language Model (LLM) into a MoE-style model that possesses capabilities in multiple specialized domains. During instruction-tuning, SIMoE automatically identifies multiple specialized experts under a specified sparsity constraint, with each expert representing a structurally sparse subset of the seed LLM's parameters that correspond to domain-specific knowledge within the data. SIMoE simultaneously learns an input-dependent expert merging strategy via a router network, leveraging rich cross-expert knowledge for superior downstream generalization that surpasses existing baselines. Empirically, SIMoE consistently achieves state-of-the-art performance on common instruction-tuning benchmarks while maintaining an optimal performance-compute trade-off compared to all baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 30 of its 47 references that are in the map. First posted on arXiv 14 Jun 2025.

Builds on

30 of 47 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Upcycling Large Language Models into Mixture of Experts10 Oct
  2. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts15 Aug
  3. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM12 Mar
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  5. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  9. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  10. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  12. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
11 more
  1. 2023Unified Concept Editing in Diffusion Modelscited by 388
  2. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  3. 2023From Sparse to Soft Mixtures of Expertscited by 248
  4. 2021Hash Layers For Large Sparse Modelscited by 237
  5. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  6. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  7. 2023Soft Merging of Experts with Adaptive Routingcited by 90
  8. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  9. 2021Learning where to learn: Gradient sparsity in meta and continual learningcited by 33
  10. 2022Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraintscited by 25
  11. 2024Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Expertscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging
  2. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
  3. 2026DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition
  4. 2026Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling
  5. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
  6. 2025DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
  7. 2025Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
  8. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs