mlsanityData to 7 Oct 2026

HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts

arXiv 2312.07035 ↗First posted 12 Dec 2023cs.LG cs.AIcited by 21

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

By routing input tokens to only a few split experts, Sparse Mixture-of-Experts has enabled efficient training of large language models. Recent findings suggest that fixing the routers can achieve competitive performance by alleviating the collapsing problem, where all experts eventually learn similar representations. However, this strategy has two key limitations: (i) the policy derived from random routers might be sub-optimal, and (ii) it requires extensive resources during training and evaluation, leading to limited efficiency gains. This work introduces HyperRout, which dynamically generates the router's parameters through a fixed hypernetwork and trainable embeddings to achieve a balance between training the routers and freezing them to learn an improved routing policy. Extensive experiments across a wide range of tasks demonstrate the superior performance and efficiency gains of HyperRouter compared to existing routing methods. Our implementation is publicly available at urlhttps://github.com/giangdip2410/HyperRouter.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 2 of its 5 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

2 of 5 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 1

  1. 2022A Review of Sparse Expert Models in Deep Learningcited by 147

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 6; 2026: 8.

21 papers.

Sort:

  1. 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Expertscited by 25
  2. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  3. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  4. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  5. 2024On the Role of Discrete Representation in Sparse Mixture of Expertscited by 4
  6. 2025Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layerscited by 4
  7. 2024SimSMoE: Solving Representational Collapse via Similarity Measurecited by 3
  8. 2025Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweepscited by 2
  9. 2025Expert Merging in Sparse Mixture of Experts with Nash Bargainingcited by 2
  10. 2026Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activationscited by 2
  11. 2024Routing in Sparsely-gated Language Models responds to Contextcited by 1
  12. 2025Rethinking Sparse Mixture of Experts from a Unified Perspectivecited by 1
9 more
  1. 2025CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via Competitioncited by 1
  2. 2025LoRALib: A Standardized Benchmark for Evaluating LoRA-MoE Methodscited by 1
  3. 2026Eigenvectors of Experts are Training-free Non-collapsing Routerscited by 1
  4. 2026Multi-level context Modeling for consistent expert selection in Mixture-of-Expertscited by 1
  5. 2026Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Expertscited by 0
  6. 2026Excitation: Momentum For Expertscited by 0
  7. 2026Do Domain-specific Experts exist in MoE-based LLMs?cited by 0
  8. 2026Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEscited by 0
  9. 2026Attention-Aware Routing: Coupling Routing and Attention in MoEscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022On the Representation Collapse of Sparse Mixture of Experts
  2. 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
  3. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Models
  4. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
  5. 2025Multilingual Routing in Mixture-of-Experts
  6. 2025Selective Sinkhorn Routing for Improved Sparse Mixture of Experts
  7. 2026Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
  8. 2026Path-Constrained Mixture-of-Experts