mlsanityData to 7 Oct 2026

DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models

arXiv 2409.06669 ↗First posted 10 Sep 2024cs.LGcited by 15

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Transformer-based Mixture-of-Experts (MoE) models have been driving several recent technological advancements in Natural Language Processing (NLP). These MoE models adopt a router mechanism to determine which experts to activate for routing input tokens. However, existing router mechanisms allocate a fixed number of experts to each token, which neglects the varying importance of different input tokens. In this study, we propose a novel dynamic router mechanism that Dynamically Allocates a variable number of experts for Mixture-of-Experts (DA-MoE) models based on an effective token importance measure. First, we show that the Transformer attention mechanism provides a natural and effective way of calculating token importance. Second, we propose a dynamic router mechanism that effectively decides the optimal number of experts (K) and allocates the top-K experts for each input token. Third, comprehensive experiments on several benchmark datasets demonstrate that our DA-MoE approach consistently outperforms the state-of-the-art Transformer based MoE model on the popular GLUE benchmark.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 45 of its 64 references that are in the map. First posted on arXiv 10 Sep 2024.

Builds on

45 of 64 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024A Survey on Mixture of Experts in Large Language Models26 Jun
  2. 2024Scaling Laws for Fine-Grained Mixture of Experts12 Feb
  3. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
27 more
  1. 2016Pointer Sentinel Mixture Modelscited by 2,992
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2018Neural Network Acceptability Judgmentscited by 952
  12. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  13. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  14. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  15. 2021A Survey of Transformerscited by 399
  16. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  17. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  18. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  19. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  20. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  21. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  22. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  23. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 124
  24. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  25. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  26. 2023Rethinking Learning Rate Tuning in the Era of Large Language Modelscited by 16
  27. 2018A Comparative Measurement Study of Deep Learning as a Service Frameworkcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 10.

15 papers.

Sort:

By area:

  1. 2025MasRouter: Learning to Route LLMs for Multi-Agent Systemscited by 80
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  3. 2025DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptationcited by 4
  4. 2026Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMscited by 4
  5. 2026Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inferencecited by 3
  6. 2026dMoE: dLLMs with Learnable Block Expertscited by 2
  7. 2025Dynamic Reasoning Chains through Depth-Specialized Mixture-of-Experts in Transformer Architecturescited by 1
  8. 2026DirMoE: Dirichlet-routed Mixture of Expertscited by 1
  9. 2026Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuningcited by 1
  10. 2026BEAM: Binary Expert Activation Masking for Dynamic Routing in MoEcited by 1
  11. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  12. 2026AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decodingcited by 1
3 more
  1. 2025AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expertcited by 0
  2. 2026ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Expertscited by 0
  3. 2026EAT: Expert Account Tracker for Efficient MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
  2. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  3. 2026A theoretical model for task routing in mixture-of-expert transformers
  4. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  5. 2025Mixture of Experts in Large Language Models
  6. 2025Advancing Expert Specialization for Better MoE
  7. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Experts
  8. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns