mlsanityData to 7 Oct 2026

Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts

arXiv 2406.12034 ↗First posted 17 Jun 2024cs.CL cs.LGcited by 22

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We present Self-MoE, an approach that transforms a monolithic LLM into a compositional, modular system of self-specialized experts, named MiXSE (MiXture of Self-specialized Experts). Our approach leverages self-specialization, which constructs expert modules using self-generated synthetic data, each equipping a shared base LLM with distinct domain-specific capabilities, activated via self-optimized routing. This allows for dynamic and capability-specific handling of various target tasks, enhancing overall capabilities, without extensive human-labeled data and added parameters. Our empirical results reveal that specializing LLMs may exhibit potential trade-offs in performances on non-specialized tasks. On the other hand, our Self-MoE demonstrates substantial improvements (6.5%p on average) over the base LLM across diverse benchmarks such as knowledge, reasoning, math, and coding. It also consistently outperforms other methods, including instance merging and weight merging, while offering better flexibility and interpretability by design with semantic experts and routing. Our findings highlight the critical role of modularity, the applicability of Self-MoE to multiple base LLMs, and the potential of self-improvement in achieving efficient, scalable, and adaptable systems.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 52 of its 67 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

52 of 67 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM12 Mar
  2. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec

Other references · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2021Program Synthesis with Large Language Modelscited by 3,509
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
37 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2022Editing Models with Task Arithmeticcited by 1,076
  9. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  10. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  11. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  12. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  13. 2021MetaICL: Learning to Learn In Contextcited by 450
  14. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  15. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  16. 2023Explainability for Large Language Models: A Surveycited by 331
  17. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  18. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  19. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  20. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  21. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  22. 2023Efficient Large Language Models: A Surveycited by 183
  23. 2024Mixture of LoRA Expertscited by 179
  24. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  25. 2023Self-Alignment with Instruction Backtranslationcited by 149
  26. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  27. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  28. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  29. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 114
  30. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  31. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  32. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  33. 2024X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Designcited by 41
  34. 2024LAB: Large-Scale Alignment for ChatBotscited by 39
  35. 2023Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Modelscited by 11
  36. 2023Empower Nested Boolean Logic via Self-Supervised Curriculum Learningcited by 7
  37. 2023Self-Specialization: Uncovering Latent Expertise within Large Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 11; 2026: 6.

22 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  3. 2025Advancing Expert Specialization for Better MoEcited by 37
  4. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEscited by 13
  5. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
  6. 2025VSC: Visual Search Compositional Text-to-Image Diffusion Modelcited by 7
  7. 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Levelcited by 7
  8. 2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Mergingcited by 6
  9. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  10. 2025RouteMark: A Fingerprint for Intellectual Property Attribution in Routing-based Model Mergingcited by 3
  11. 2025SMAGDi: Socratic Multi Agent Interaction Graph Distillation for Efficient High Accuracy Reasoningcited by 3
  12. 2026Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretrainingcited by 3
10 more
  1. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Modelscited by 2
  2. 2024Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Expertscited by 2
  3. 2025Feedback-Driven Vision-Language Alignment with Minimal Human Supervisioncited by 1
  4. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Expertscited by 1
  5. 2026Skill Weaving: Efficient LLM Improvement via Modular Skillpackscited by 1
  6. 2025Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectivenesscited by 0
  7. 2025Agentic Meta-Orchestrator for Multi-task Copilotscited by 0
  8. 2026Learning to Recover Task Experts from a Multi-Task Merged Modelcited by 0
  9. 2026PrimeScientist: Strategic Allocation of Research Effort in Autonomous Researchcited by 0
  10. 2026CORTEX: Learning to Share and Specialize in Dense Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM Augmented LLMs: Expanding Capabilities through Composition
  2. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
  3. 2026Do Domain-specific Experts exist in MoE-based LLMs?
  4. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
  5. 2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment
  6. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  7. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
  8. 2021Efficient Large Scale Language Modeling with Mixtures of Experts