mlsanityData to 7 Oct 2026

Multi-Head Mixture-of-Experts

arXiv 2404.15045 ↗First posted 23 Apr 2024cs.CL cs.AI cs.LGcited by 30

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Sparse Mixtures of Experts (SMoE) scales model capacity without significant increases in training and inference costs, but exhibits the following two issues: (1) Low expert activation, where only a small subset of experts are activated for optimization. (2) Lacking fine-grained analytical capabilities for multiple semantic concepts within individual tokens. We propose Multi-Head Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each token into multiple sub-tokens. These sub-tokens are then assigned to and processed by a diverse set of experts in parallel, and seamlessly reintegrated into the original token form. The multi-head mechanism enables the model to collectively attend to information from various representation spaces within different experts, while significantly enhances expert activation, thus deepens context understanding and alleviate overfitting. Moreover, our MH-MoE is straightforward to implement and decouples from other SMoE optimization methods, making it easy to integrate with other SMoE models for enhanced performance. Extensive experimental results across three tasks: English-focused language modeling, Multi-lingual language modeling and Masked multi-modality modeling tasks, demonstrate the effectiveness of MH-MoE.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 32 of its 42 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

32 of 42 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Mixtral of Experts8 Jan
  2. 2023Task-Based MoE for Multitask Multilingual Machine Translation30 Aug
  3. 2022MoEC: Mixture of Expert Clusters19 Jul
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
14 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  7. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  8. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  9. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  10. 2018A Simple Method for Commonsense Reasoningcited by 223
  11. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  12. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  13. 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
  14. 2021Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognitioncited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 11; 2026: 12.

30 papers.

Sort:

By area:

  1. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 56
  2. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  3. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  4. 2025Mixture of Experts in Large Language Modelscited by 15
  5. 2025TimeExpert: An Expert-Guided Video LLM for Video Temporal Groundingcited by 14
  6. 2024Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Visioncited by 13
  7. 2024Learning More Generalized Experts by Merging Experts in Mixture-of-Expertscited by 12
  8. 2025UMoE: Unifying Attention and FFN with Shared Expertscited by 6
  9. 2025A Knowledge-Driven Diffusion Policy for End-to-End Autonomous Driving Based on Expert Routingcited by 5
  10. 2024A Dynamic Approach to Stock Price Prediction: Comparing RNN and Mixture of Experts Models Across Different Volatility Profilescited by 4
  11. 2024MH-MoE: Multi-Head Mixture-of-Expertscited by 4
  12. 2025Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspectivecited by 3
18 more
  1. 2026Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Expertscited by 3
  2. 2024Mixture of Hidden-Dimensions Transformercited by 2
  3. 2025BLR-MoE: Boosted Language-Routing Mixture of Experts for Domain-Robust Multilingual E2E ASRcited by 2
  4. 2025How Do Consumers Really Choose: Exposing Hidden Preferences with the Mixture of Experts Modelcited by 2
  5. 2025Online Mixture of Experts: No-Regret Learning for Optimal Collective Decision-Makingcited by 2
  6. 2026Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelismcited by 2
  7. 2025Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcyclingcited by 1
  8. 2025Flash Multi-Head Feed-Forward Networkcited by 1
  9. 2026L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Expertscited by 1
  10. 2026AME-TS: Anchored Mixture-of-Experts for Time Series Forecastingcited by 1
  11. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  12. 2026Multi-Mixer Models: Flexible Sequence Modeling with Shared Representationscited by 1
  13. 2026Counterfactual Understanding via Retrieval-aware Multimodal Modeling for Time-to-Event Survival Predictioncited by 0
  14. 2026FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approachcited by 0
  15. 2026ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Expertscited by 0
  16. 2026A Multi-task Mixture of Experts Framework for Malware Classification, Packing Detection, and Family Attributioncited by 0
  17. 2026VGGT-Prime: Compute-Adaptive Mixture-of-Heads for Efficient Visual Geometry Transformerscited by 0
  18. 2026Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  2. 2024HMoE: Heterogeneous Mixture of Experts for Language Modeling
  3. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
  4. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  5. 2025Beyond Redundancy: Diverse and Specialized Multi-Expert Sparse Autoencoder
  6. 2024A Survey on Mixture of Experts in Large Language Models
  7. 2025Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models
  8. 2024A Closer Look into Mixture-of-Experts in Large Language Models