mlsanityData to 7 Oct 2026

Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning

arXiv 2509.10513 ↗First posted 3 Sep 2025cs.LGcited by 3

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

A sparse Mixture-of-Experts (MoE) architecture has emerged as a highly scalable solution by conditionally activating sub-modules without a proportional increase in computational costs. However, improving expert specialization to enhance performance and generalization remains a challenge for MoE, especially in instruction tuning scenarios characterized by significant input heterogeneity. In this work, we propose the Mixture-of-Clustered-Experts (MoCE) to address this limitation through a dual-stage routing mechanism. The first stage in the mechanism performs expert group routing based on sequence-level features, while the second stage activates the top-k experts within the group at the token level. This approach enables the effective partitioning of heterogeneous inputs based on their knowledge requirements, encouraging expert group specialization while maintaining the advantages of token-level routing. We evaluate MoCE across a comprehensive set of benchmarks, demonstrating its consistent superiority over strong baselines and its enhanced generalization capabilities. Detailed analysis further highlights the robustness and effectiveness of MoCE.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 35 of its 51 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

35 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Soft Merging of Experts with Adaptive Routing6 Jun
  2. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers2 Mar
  3. 2022Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts8 Oct

Other references · 32

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  10. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
20 more
  1. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  2. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  3. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  5. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  6. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  7. 2023From Sparse to Soft Mixtures of Expertscited by 245
  8. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  9. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  10. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  11. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  12. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  13. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  14. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  15. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  16. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
  17. 2022One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Codecited by 26
  18. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Taskscited by 22
  19. 2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Expertscited by 19
  20. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlapcited by 0
  2. 2026RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanismcited by 0
  3. 2026Router Prior Bias: Preserving Base Routing Structure in MoE Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
  2. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
  3. 2025CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning
  4. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimization
  5. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  6. 2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging
  7. 2026MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning
  8. 2026IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts