mlsanityData to 7 Oct 2026

Adaptive Gating in Mixture-of-Experts based Language Models

arXiv 2310.07188 ↗First posted 11 Oct 2023cs.CL cs.AIcited by 16

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large language models, such as OpenAI's ChatGPT, have demonstrated exceptional language understanding capabilities in various NLP tasks. Sparsely activated mixture-of-experts (MoE) has emerged as a promising solution for scaling models while maintaining a constant number of computational operations. Existing MoE model adopts a fixed gating network where each token is computed by the same number of experts. However, this approach contradicts our intuition that the tokens in each sequence vary in terms of their linguistic complexity and, consequently, require different computational costs. Little is discussed in prior research on the trade-off between computation per token and model performance. This paper introduces adaptive gating in MoE, a flexible training strategy that allows tokens to be processed by a variable number of experts based on expert probability distribution. The proposed framework preserves sparsity while improving training efficiency. Additionally, curriculum learning is leveraged to further reduce training time. Extensive experiments on diverse NLP tasks show that adaptive gating reduces at most 22.5% training time while maintaining inference quality. Moreover, we conduct a comprehensive analysis of the routing decisions and present our insights when adaptive gating is used.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 29 of its 39 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

29 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  3. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  9. 2015Teaching Machines to Read and Comprehendcited by 1,039
  10. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  12. 2022Efficiently Scaling Transformer Inferencecited by 529
13 more
  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  2. 2021Hash Layers For Large Sparse Modelscited by 236
  3. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  4. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  5. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  6. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  7. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  8. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  9. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  10. 2023MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognitioncited by 31
  11. 2022One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Codecited by 26
  12. 2022One Student Knows All Experts Know: From Sparse to Densecited by 25
  13. 2021Heterogeneous Multi-task Learning with Expert Diversitycited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 6.

16 papers.

Sort:

By area:

  1. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
  2. 2024AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Expertscited by 33
  3. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  4. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inferencecited by 16
  5. 2025Cooperation of Experts: Fusing Heterogeneous Information with Large Margincited by 6
  6. 2024LaDiMo: Layer-wise Distillation Inspired MoEfiercited by 4
  7. 2025CoT-VLM4Tar: Chain-of-Thought Guided Vision-Language Models for Traffic Anomaly Resolutioncited by 3
  8. 2025Interpretable Few-Shot Image Classification via Prototypical Concept-Guided Mixture of LoRA Expertscited by 1
  9. 2026Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervisioncited by 1
  10. 2026BEAM: Binary Expert Activation Masking for Dynamic Routing in MoEcited by 1
  11. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  12. 2024Adaptive Inference: Theoretical Limits and Unexplored Opportunitiescited by 0
4 more
  1. 2025Nav-EE: Navigation-Guided Early Exiting for Efficient Vision-Language Models in Autonomous Drivingcited by 0
  2. 2026When Adaptation Fails: A Gradient-Based Diagnosis of Collapsed Gating in Vision-Language Prompt Learningcited by 0
  3. 2026MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Modelscited by 0
  4. 2026BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Experts in Large Language Models
  2. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  3. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  4. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
  5. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  6. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
  7. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  8. 2024A Survey on Mixture of Experts in Large Language Models