mlsanityData to 7 Oct 2026

MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models

arXiv 2512.12121 ↗First posted 13 Dec 2025cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

We introduce MixtureKit, a modular open-source framework for constructing, training, and analyzing Mixture-of-Experts (MoE) models from arbitrary pre-trained or fine-tuned models. MixtureKit currently supports three complementary methods: (i) Traditional MoE, which uses a single router per transformer block to select experts, (ii) BTX (Branch-Train-Mix), which introduces separate routers for each specified sub-layer enabling fine-grained token routing, and (iii) BTS (Branch-Train-Stitch), which keeps experts fully intact and introduces trainable stitch layers for controlled information exchange between hub and experts. MixtureKit automatically modifies the model configuration, patches decoder and causal LM classes, and saves a unified checkpoint ready for inference or fine-tuning. We further provide a visualization interface to inspect per-token routing decisions, expert weight distributions, and layer-wise contributions. Experiments with multilingual code-switched data (e.g. Arabic-Latin) show that a BTX-based model trained using MixtureKit can outperform baseline dense models on multiple benchmarks. We release MixtureKit as a practical foundation for research and development of MoE-based systems across diverse domains.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 21 of its 30 references that are in the map. First posted on arXiv 13 Dec 2025.

Builds on

21 of 30 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications10 Mar

Other references · 20

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  6. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  7. 2023BloombergGPT: A Large Language Model for Financecited by 923
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  9. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  10. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  11. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  12. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
8 more
  1. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  2. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  3. 2024The rising costs of training frontier AI modelscited by 91
  4. 2023Exploring the Carbon Footprint of Hugging Face's ML Models: A Repository Mining Studycited by 23
  5. 2024The AI Community Building the Future? A Quantitative Analysis of Development Activity on Hugging Face Hubcited by 18
  6. 2025Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Taskscited by 8
  7. 2023Statistically Valid Variable Importance Assessment through Conditional Permutationscited by 7
  8. 2025Nile-Chat: Egyptian Language Models for Arabic and Latin Scriptscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Revisiting Single-gated Mixtures of Experts
  2. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
  3. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
  4. 2025Bayesian Mixture-of-Experts: Towards Making LLMs Know What They Don't Know
  5. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  6. 2026MoE Lens -- An Expert Is All You Need
  7. 2026Towards a Statistical Understanding of Mixture-of-Experts
  8. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models