mlsanityData to 14 Oct 2026

Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models

arXiv 2405.14297 ↗First posted 23 May 2024cs.LG cs.AIcited by 56

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

The Sparse Mixture of Experts (SMoE) has been widely employed to enhance the efficiency of training and inference for Transformer-based foundational models, yielding promising results.However, the performance of SMoE heavily depends on the choice of hyper-parameters, such as the number of experts and the number of experts to be activated (referred to as top-k), resulting in significant computational overhead due to the extensive model training by searching over various hyper-parameter configurations. As a remedy, we introduce the Dynamic Mixture of Experts (DynMoE) technique. DynMoE incorporates (1) a novel gating method that enables each token to automatically determine the number of experts to activate. (2) An adaptive process automatically adjusts the number of experts during training. Extensive numerical results across Vision, Language, and Vision-Language tasks demonstrate the effectiveness of our approach to achieve competitive performance compared to GMoE for vision and language tasks, and MoE-LLaVA for vision-language tasks, while maintaining efficiency by activating fewer parameters. Our code is available at https://github.com/LINs-lab/DynMoE.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 52 of its 69 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

52 of 69 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Models12 Mar
  2. 2024Towards an empirical understanding of MoE design choices20 Feb
  3. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  4. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Experts5 Oct
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Segment Anythingcited by 8,234
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022Scalable Diffusion Models with Transformerscited by 6,054
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
35 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  6. 2024Mixtral of Expertscited by 1,823
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  12. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  13. 2020In Search of Lost Domain Generalizationcited by 1,054
  14. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  15. 2018Neural Network Acceptability Judgmentscited by 958
  16. 2023A Survey on Multimodal Large Language Modelscited by 909
  17. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  18. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  19. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  20. 2021Dynamic Neural Networks: A Surveycited by 408
  21. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  22. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  23. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  24. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  25. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  26. 2023From Sparse to Soft Mixtures of Expertscited by 248
  27. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  28. 2019Generalizing to unseen domains via distribution matchingcited by 110
  29. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  30. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Visioncited by 67
  31. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 56
  32. 2024Multi-Head Mixture-of-Expertscited by 31
  33. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectioncited by 28
  34. 2024Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptationcited by 25
  35. 2024Learn To be Efficient: Build Structured Sparsity in Large Language Modelscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 22; 2026: 27.

56 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  3. 2025Advancing Expert Specialization for Better MoEcited by 39
  4. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
  5. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 14
  6. 2025Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Expertscited by 14
  7. 2025TimeExpert: An Expert-Guided Video LLM for Video Temporal Groundingcited by 14
  8. 2025LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Expertscited by 13
  9. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  10. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skippingcited by 11
  11. 2025HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policiescited by 10
  12. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Expertscited by 9
38 more
  1. 2025Tight Clusters Make Specialized Expertscited by 8
  2. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Serverscited by 8
  3. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Modelcited by 7
  4. 2025Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilizationcited by 7
  5. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inferencecited by 6
  6. 2025Route Experts by Sequence, not by Tokencited by 6
  7. 2025Large Language Model Sourcing: A Surveycited by 5
  8. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specializationcited by 5
  9. 2024LaDiMo: Layer-wise Distillation Inspired MoEfiercited by 4
  10. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  11. 2025How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Expertscited by 4
  12. 2026Post-Trained MoE Can Skip Half Experts via Self-Distillationcited by 4
  13. 2024Expert-Token Resonance MoE: Bidirectional Routing with Efficiency Affinity-Driven Active Selectioncited by 3
  14. 2025DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-trainingcited by 3
  15. 2026Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancingcited by 3
  16. 2026Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learningcited by 2
  17. 2026On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Modelscited by 2
  18. 2025Cluster Topology-Driven Placement of Experts Reduces Network Traffic in MoE Inferencecited by 1
  19. 2025MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designscited by 1
  20. 2026DirMoE: Dirichlet-routed Mixture of Expertscited by 1
  21. 2026BEAM: Binary Expert Activation Masking for Dynamic Routing in MoEcited by 1
  22. 2026RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanismcited by 1
  23. 2026ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Expertscited by 1
  24. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  25. 2025A global log for medical AIcited by 0
  26. 2025Self-Adaptive Graph Mixture of Modelscited by 0
  27. 2025OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latencycited by 0
  28. 2026Compressing LLMs with MoP: Mixture of Prunerscited by 0
  29. 2026Evolutionary Router Feature Generation for Zero-Shot Graph Anomaly Detection with Mixture-of-Expertscited by 0
  30. 2026One Model to Translate Them All: Universal Any-to-Any Translation for Heterogeneous Collaborative Perceptioncited by 0
  31. 2026PithTrain: A Compact and Agent-Native MoE Training Systemcited by 0
  32. 2026STAR: Rethinking MoE Routing as Structure-Aware Subspace Learningcited by 0
  33. 2026Learning Subset-Shared Invariances for Domain Generalization with Mixture-of-Expertscited by 0
  34. 2026GeMoE: Gating Entropy is All You Need for Uncertainty-aware Adaptive Routing in MoE-based Large Vision-Language Modelscited by 0
  35. 2026Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selectioncited by 0
  36. 2026Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computationcited by 0
  37. 2026Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEscited by 0
  38. 2026ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMscited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gate
  2. 2025Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems
  3. 2021Taming Sparsely Activated Transformer with Stochastic Experts
  4. 2023Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity
  5. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion
  6. 2024Mixture of A Million Experts
  7. 2025Rethinking Sparse Mixture of Experts from a Unified Perspective
  8. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers