mlsanityData to 7 Oct 2026

HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts

arXiv 2402.12656 ↗First posted 20 Feb 2024cs.LG cs.AIcited by 25

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The Mixture of Experts (MoE) for language models has been proven effective in augmenting the capacity of models by dynamically routing each input token to a specific subset of experts for processing. Despite the success, most existing methods face a challenge for balance between sparsity and the availability of expert knowledge: enhancing performance through increased use of expert knowledge often results in diminishing sparsity during expert selection. To mitigate this contradiction, we propose HyperMoE, a novel MoE framework built upon Hypernetworks. This framework integrates the computational processes of MoE with the concept of knowledge transferring in multi-task learning. Specific modules generated based on the information of unselected experts serve as supplementary information, which allows the knowledge of experts not selected to be used while maintaining selection sparsity. Our comprehensive empirical evaluations across multiple datasets and backbones establish that HyperMoE significantly outperforms existing MoE methods under identical conditions concerning the number of experts.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 57 of its 75 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

57 of 75 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models29 Jan
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  3. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Experts15 Oct
  4. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2016Pointer Sentinel Mixture Modelscited by 2,992
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  7. 2015Teaching Machines to Read and Comprehendcited by 1,039
  8. 2016HyperNetworkscited by 1,003
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  10. 2018Neural Network Acceptability Judgmentscited by 952
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  12. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  13. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  14. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  15. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  16. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  17. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  18. 2021Hash Layers For Large Sparse Modelscited by 236
  19. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  20. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  21. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  22. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 98
  23. 2023Modular Deep Learningcited by 82
  24. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  25. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  26. 2023Massive Editing for Large Language Models via Meta Learningcited by 62
  27. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 45
  28. 2022HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisationcited by 28
  29. 2020Parameter Space Factorization for Zero-Shot Learning across Tasks and Languagescited by 22
  30. 2022Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfercited by 22
  31. 2022Hyperdecoders: Instance-specific decoders for multi-task NLPcited by 21
  32. 2023HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Expertscited by 21
  33. 2022Example-based Hypernetworks for Out-of-Distribution Generalizationcited by 16
  34. 2022Multilingual Machine Translation with Hyper-Adapterscited by 15
  35. 2022Boosting Natural Language Generation from Instructions with Meta-Learningcited by 15
  36. 2021Character-level HyperNetworks for Hate Speech Detectioncited by 7
  37. 2023Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuningcited by 6
  38. 2024Read to Play (R2-Play): Decision Transformer with Multimodal Game Instructioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 11; 2026: 5.

25 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  3. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 26
  4. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruningcited by 24
  5. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  6. 2025Mixture of Experts in Large Language Modelscited by 15
  7. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
  8. 2024MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Expertscited by 9
  9. 2024GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theorycited by 8
  10. 2024PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQLcited by 7
  11. 2024On the Risk of Evidence Pollution for Malicious Social Text Detection in the Era of LLMscited by 5
  12. 2024CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Expertscited by 5
13 more
  1. 2025EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Modelscited by 5
  2. 2024Mixture of Diverse Size Expertscited by 4
  3. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognitioncited by 4
  4. 2025THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translationcited by 2
  5. 2026Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinationscited by 2
  6. 2025COME: Dual Structure-Semantic Learning with Collaborative MoE for Universal Lesion Detection Across Heterogeneous Ultrasound Datasetscited by 1
  7. 2025VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reusecited by 1
  8. 2026SDG-MoE: Signed Debate Graph Mixture-of-Expertscited by 1
  9. 2025Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Onescited by 0
  10. 2025Ask, Answer, and Detect: Role-Playing LLMs for Personality Detection with Question-Conditioned Mixture-of-Expertscited by 0
  11. 2026SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignmentcited by 0
  12. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Datacited by 0
  13. 2026IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models
  2. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  3. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  4. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
  5. 2025GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
  6. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts
  7. 2025Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization
  8. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models