mlsanityData to 7 Oct 2026

MH-MoE: Multi-Head Mixture-of-Experts

arXiv 2411.16205 ↗First posted 25 Nov 2024cs.CLcited by 4

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by using the multi-head mechanism to collectively attend to information from various representation spaces within different experts. In this paper, we present a novel implementation of MH-MoE that maintains both FLOPs and parameter parity with sparse Mixture of Experts models. Experimental results on language models show that the new implementation yields quality improvements over both vanilla MoE and fine-grained MoE models. Additionally, our experiments demonstrate that MH-MoE is compatible with 1-bit Large Language Models (LLMs) such as BitNet.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 12 of its 17 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

12 of 17 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Multi-Head Mixture-of-Experts23 Apr
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  3. 2023Task-Based MoE for Multitask Multilingual Machine Translation30 Aug
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 6

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020GLU Variants Improve Transformercited by 1,724
  3. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  4. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  5. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  6. 2021Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognitioncited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelismcited by 2
  2. 2025Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcyclingcited by 1
  3. 2026CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answeringcited by 1
  4. 2026MixTTE: Multi-Level Mixture-of-Experts for Scalable and Adaptive Travel Time Estimationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Experts in Large Language Models
  2. 2026Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
  3. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models
  4. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  5. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
  6. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness
  7. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More
  8. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models