mlsanityData to 14 Oct 2026

Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts

arXiv 2408.17280 ↗First posted 30 Aug 2024cs.AI cs.CLcited by 2

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We present a toolkit for creating low-cost Mixture-of-Domain-Experts (MOE) from trained models. The toolkit can be used for creating a mixture from models or from adapters. We perform extensive tests and offer guidance on defining the architecture of the resulting MOE using the toolkit. A public repository is available.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 18 of its 30 references that are in the map. First posted on arXiv 30 Aug 2024.

Builds on

18 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts17 Jun
  2. 2024Mixtral of Experts8 Jan

Other references · 16

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  8. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  9. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  10. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  11. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  12. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
4 more
  1. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  2. 2024A Survey of Large Language Models in Finance (FinLLMs)cited by 79
  3. 2024LLM Augmented LLMs: Expanding Capabilities through Compositioncited by 35
  4. 2024What the Weight?! A Unified Framework for Zero-Shot Knowledge Compositioncited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025Mixture of Experts in Large Language Modelscited by 15
  2. 2024Collaboratively adding new knowledge to an LLMcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Mixture of Experts in Large Language Models
  2. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  3. 2026Do Domain-specific Experts exist in MoE-based LLMs?
  4. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  5. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts
  6. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
  7. 2025Advancing Expert Specialization for Better MoE
  8. 2026UMoE:Unlocking Every Expert in Domain-Specific Training