mlsanityData to 7 Oct 2026

Harder Tasks Need More Experts: Dynamic Routing in MoE Models

arXiv 2403.07652 ↗First posted 12 Mar 2024cs.LG cs.CLcited by 71

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

In this paper, we introduce a novel dynamic expert selection framework for Mixture of Experts (MoE) models, aiming to enhance computational efficiency and model performance by adjusting the number of activated experts based on input difficulty. Unlike traditional MoE approaches that rely on fixed Top-K routing, which activates a predetermined number of experts regardless of the input's complexity, our method dynamically selects experts based on the confidence level in expert selection for each input. This allows for a more efficient utilization of computational resources, activating more experts for complex tasks requiring advanced reasoning and fewer for simpler tasks. Through extensive evaluations, our dynamic routing method demonstrates substantial improvements over conventional Top-2 routing across various benchmarks, achieving an average improvement of 0.7% with less than 90% activated parameters. Further analysis shows our model dispatches more experts to tasks requiring complex reasoning skills, like BBH, confirming its ability to dynamically allocate computational resources in alignment with the input's complexity. Our findings also highlight a variation in the number of experts needed across different layers of the transformer model, offering insights into the potential for designing heterogeneous MoE frameworks. The code and models are available at https://github.com/ZhenweiAn/Dynamic_MoE.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 22 of its 28 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

22 of 28 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  2. 2021Taming Sparsely Activated Transformer with Stochastic Experts8 Oct
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 19

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2020GLU Variants Improve Transformercited by 1,724
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  11. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  12. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
7 more
  1. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  2. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  3. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  4. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  5. 2021Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challengecited by 79
  6. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  7. 2020Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inferencecited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 31; 2026: 21.

71 papers.

Sort:

By area:

  1. 2025Multi-agent Architecture Search via Agentic Supernetcited by 164
  2. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  3. 2025MasRouter: Learning to Route LLMs for Multi-Agent Systemscited by 80
  4. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  5. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  6. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 28
  7. 2025Difficulty-Aware Agentic Orchestration for Query-Specific Multi-Agent Workflowscited by 23
  8. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  9. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  10. 2024Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrastcited by 19
  11. 2024Ada-K Routing: Boosting the Efficiency of MoE-based LLMscited by 16
  12. 2026Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gaincited by 16
38 more
  1. 2025TimeExpert: An Expert-Guided Video LLM for Video Temporal Groundingcited by 14
  2. 2024WDMoE: Wireless Distributed Mixture of Experts for Large Language Modelscited by 12
  3. 2025Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUscited by 12
  4. 2025LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Expertscited by 12
  5. 2024Temporal Scaling Law for Large Language Modelscited by 11
  6. 2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanismcited by 10
  7. 2024MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Expertscited by 9
  8. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Expertscited by 9
  9. 2024GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theorycited by 8
  10. 2026Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Expertscited by 8
  11. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  12. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Expertscited by 6
  13. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Modelcited by 6
  14. 2025BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsitycited by 6
  15. 2025Extracting Uncertainty Estimates from Mixtures of Experts for Semantic Segmentationcited by 6
  16. 2025Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilizationcited by 6
  17. 2025Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retrainingcited by 6
  18. 2024CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Expertscited by 5
  19. 2025Route Experts by Sequence, not by Tokencited by 5
  20. 2024LaDiMo: Layer-wise Distillation Inspired MoEfiercited by 4
  21. 2024Glider: Global and Local Instruction-Driven Expert Routercited by 4
  22. 2024Expert-Token Resonance MoE: Bidirectional Routing with Efficiency Affinity-Driven Active Selectioncited by 3
  23. 2024From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuningcited by 3
  24. 2025Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilitiescited by 3
  25. 2025Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compressioncited by 3
  26. 2025Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Modelscited by 3
  27. 2026Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancingcited by 3
  28. 2026Learning Graph Foundation Models on Riemannian Graph-of-Graphscited by 3
  29. 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsitycited by 2
  30. 2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuningcited by 2
  31. 2025Resource-efficient Inference with Foundation Model Programscited by 2
  32. 2025Mixture-of-Experts for Personalized and Semantic-Aware Next Location Predictioncited by 2
  33. 2025FastMMoE: Accelerating Multimodal Large Language Models through Dynamic Expert Activation and Routing-Aware Token Pruningcited by 2
  34. 2025DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-trainingcited by 2
  35. 2026Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learningcited by 2
  36. 2026Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifoldscited by 2
  37. 2026Compressed Sensing for Capability Localization in Large Language Modelscited by 2
  38. 2026TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decompositioncited by 2

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills
  2. 2025Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
  3. 2026Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
  4. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
  5. 2026MoE Lens -- An Expert Is All You Need
  6. 2026MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning
  7. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Models
  8. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale