mlsanityData to 7 Oct 2026

MoECollab: Democratizing LLM Development Through Collaborative Mixture of Experts

arXiv 2503.12592 ↗First posted 16 Mar 2025cs.LG cs.AI cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large Language Model (LLM) development has become increasingly centralized, limiting participation to well-resourced organizations. This paper introduces MoECollab, a novel framework leveraging Mixture of Experts (MoE) architecture to enable distributed, collaborative LLM development. By decomposing monolithic models into specialized expert modules coordinated by a trainable gating network, our framework allows diverse contributors to participate regardless of computational resources. We provide a complete technical implementation with mathematical foundations for expert dynamics, gating mechanisms, and integration strategies. Experiments on multiple datasets demonstrate that our approach achieves accuracy improvements of 3-7% over baseline models while reducing computational requirements by 34%. Expert specialization yields significant domain-specific gains, with improvements from 51% to 88% F1 score in general classification and from 23% to 44% accuracy in news categorization. We formalize the routing entropy optimization problem and demonstrate how proper regularization techniques lead to 14% higher expert utilization rates. These results validate MoECollab as an effective approach for democratizing LLM development through architecturally-supported collaboration.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 6 of its 10 references that are in the map. First posted on arXiv 16 Mar 2025.

Builds on

6 of 10 references.

References · 6

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  5. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  6. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing Expert Specialization for Better MoE
  2. 2026Do Domain-specific Experts exist in MoE-based LLMs?
  3. 2026Token-Level LLM Collaboration via FusionRoute
  4. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
  5. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  6. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  7. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
  8. 2025Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression