mlsanityData to 14 Oct 2026

MoIN: Mixture of Introvert Experts to Upcycle an LLM

arXiv 2410.09687 ↗First posted 13 Oct 2024cs.LG cs.AI cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The goal of this paper is to improve (upcycle) an existing large language model without the prohibitive requirements of continued pre-training of the full-model. The idea is to split the pre-training data into semantically relevant groups and train an expert on each subset. An expert takes the form of a lightweight adapter added on the top of a frozen base model. During inference, an incoming query is first routed to the most relevant expert which is then loaded onto the base model for the forward pass. Unlike typical Mixture of Experts (MoE) models, the experts in our method do not work with other experts for a single query. Hence, we dub them "introvert" experts. Freezing the base model and keeping the experts as lightweight adapters allows extreme parallelism during training and inference. Training of all experts can be done in parallel without any communication channels between them. Similarly, the inference can also be heavily parallelized by distributing experts on different GPUs and routing each request to the GPU containing its relevant expert. We implement a proof-of-concept version of this method and show the validity of our approach.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 30 of its 38 references that are in the map. First posted on arXiv 13 Oct 2024.

Builds on

30 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  2. 2024Mixtral of Experts8 Jan
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 27

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  10. 2023StarCoder: may the source be with you!cited by 799
  11. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  12. 2021Knowledge Neurons in Pretrained Transformerscited by 601
15 more
  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  3. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  4. 2023Continual Pre-training of Language Modelscited by 198
  5. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  6. 2019Large Memory Layers with Product Keyscited by 124
  7. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 119
  8. 2013Deep Learning of Representations: Looking Forwardcited by 84
  9. 2023Punica: Multi-Tenant LoRA Servingcited by 83
  10. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  11. 2024Mixture of A Million Expertscited by 72
  12. 2023NOLA: Compressing LoRA using Linear Combination of Random Basiscited by 29
  13. 2014Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learningcited by 24
  14. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  15. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Models
  2. 2026Training-Free Dynamic Upcycling of Expert Language Models
  3. 2024Upcycling Large Language Models into Mixture of Experts
  4. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  5. 2024Mixture of Diverse Size Experts
  6. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
  7. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  8. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models