mlsanityData to 14 Oct 2026

HMoE: Heterogeneous Mixture of Experts for Language Modeling

arXiv 2408.10681 ↗First posted 20 Aug 2024cs.CL cs.LGcited by 29

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture of Experts (MoE) offers remarkable performance and computational efficiency by selectively activating subsets of model parameters. Traditionally, MoE models use homogeneous experts, each with identical capacity. However, varying complexity in input data necessitates experts with diverse capabilities, while homogeneous MoE hinders effective expert specialization and efficient parameter utilization. In this study, we propose a novel Heterogeneous Mixture of Experts (HMoE), where experts differ in size and thus possess diverse capacities. This heterogeneity allows for more specialized experts to handle varying token complexities more effectively. To address the imbalance in expert activation, we propose a novel training objective that encourages the frequent activation of smaller experts, enhancing computational efficiency and parameter utilization. Extensive experiments demonstrate that HMoE achieves lower loss with fewer activated parameters and outperforms conventional homogeneous MoE models on various pre-training evaluation benchmarks. Codes will be released upon acceptance.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 25 of its 43 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

25 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
10 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  5. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  6. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  7. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  8. 2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directionscited by 75
  9. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 72
  10. 2023Improving Expert Specialization in Mixture of Expertscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 13; 2026: 11.

29 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 90
  3. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  4. 2025Mixture of Experts in Large Language Modelscited by 15
  5. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Expertscited by 11
  6. 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 10
  7. 2024Lossless KV Cache Compression to 2%cited by 7
  8. 2025Autonomy-of-Experts Modelscited by 7
  9. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inferencecited by 6
  10. 2024Mixture of Diverse Size Expertscited by 5
  11. 2025From Dense to Dynamic: Token-Difficulty Driven MoEfication of Pre-Trained LLMscited by 5
  12. 2026DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Modelscited by 5
17 more
  1. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specializationcited by 5
  2. 2026φ-Balancing for Mixture-of-Experts Trainingcited by 5
  3. 2025HDEE: Heterogeneous Domain Expert Ensemblecited by 4
  4. 2024Sparse Checkpointing for Fast and Reliable MoE Trainingcited by 3
  5. 2025MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 3
  6. 2026NESTOR: A Nested MOE-based Neural Operator for Large-Scale PDE Pre-Trainingcited by 3
  7. 2025THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translationcited by 2
  8. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  9. 2026Mixture of Heterogeneous Grouped Experts for Language Modelingcited by 2
  10. 2025Mixture of Group Experts for Learning Invariant Representationscited by 1
  11. 2025Multi-modal Collaborative Optimization and Expansion Network for Event-assisted Single-eye Expression Recognitioncited by 0
  12. 2025AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decompositioncited by 0
  13. 2026Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneitycited by 0
  14. 2026n-Musketeers: Reinforcement Learning Shapes Collaboration Among Language Modelscited by 0
  15. 2026Universal Pansharpening Modelcited by 0
  16. 2026Task-Aware Federated Fine-Tuning for MoE-based Large Language Modelscited by 0
  17. 2026Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-Head Mixture-of-Experts
  2. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  3. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
  4. 2024A Survey on Mixture of Experts in Large Language Models
  5. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
  6. 2025Advancing Expert Specialization for Better MoE
  7. 2024Scaling Laws for Fine-Grained Mixture of Experts
  8. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models