mlsanityData to 7 Oct 2026

Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers

arXiv 2205.14336 ↗First posted 28 May 2022cs.LGcited by 22

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Sparsely activated transformers, such as Mixture of Experts (MoE), have received great interest due to their outrageous scaling capability which enables dramatical increases in model size without significant increases in computational cost. To achieve this, MoE models replace the feedforward sub-layer with Mixture-of-Experts sub-layer in transformers and use a gating network to route each token to its assigned experts. Since the common practice for efficient training of such models requires distributing experts and tokens across different machines, this routing strategy often incurs huge cross-machine communication cost because tokens and their assigned experts likely reside in different machines. In this paper, we propose Gating Dropout, which allows tokens to ignore the gating network and stay at their local machines, thus reducing the cross-machine communication. Similar to traditional dropout, we also show that Gating Dropout has a regularization effect during training, resulting in improved generalization performance. We validate the effectiveness of Gating Dropout on multilingual machine translation tasks. Our results demonstrate that Gating Dropout improves a state-of-the-art MoE model with faster wall-clock time convergence rates and better BLEU scores for a variety of model sizes and datasets.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 34 of its 44 references that are in the map. First posted on arXiv 28 May 2022.

Builds on

34 of 44 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2016Deep Networks with Stochastic Depthcited by 973
19 more
  1. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  2. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  3. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  4. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  5. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  6. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  7. 2021Hash Layers For Large Sparse Modelscited by 236
  8. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  9. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  10. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  11. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  12. 2017Curriculum Dropoutcited by 50
  13. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  14. 2020Multi-task Learning for Multilingual Neural Machine Translationcited by 46
  15. 2020Dropout: Explicit Forms and Capacity Controlcited by 27
  16. 2016Recurrent Dropout without Memory Losscited by 22
  17. 2021M6-T: Exploring Sparse Expert Models and Beyondcited by 22
  18. 2020Learnable Bernoulli Dropout for Bayesian Deep Learningcited by 14
  19. 2022Transformer with Memory Replaycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 5; 2024: 6; 2025: 6; 2026: 1.

22 papers.

Sort:

By area:

  1. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  2. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  3. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  4. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Expertscited by 46
  5. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  6. 2024Frequency Decoupling for Motion Magnification via Multi-Level Isomorphic Architecturecited by 28
  7. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 24
  8. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMscited by 22
  9. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  10. 2022MoEC: Mixture of Expert Clusterscited by 14
  11. 2022AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translationcited by 9
  12. 2022Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translationcited by 8
10 more
  1. 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedulescited by 7
  2. 2023Task-Based MoE for Multitask Multilingual Machine Translationcited by 5
  3. 2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
  4. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Taskscited by 4
  5. 2025HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swapcited by 4
  6. 2023Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacitycited by 3
  7. 2024Routing in Sparsely-gated Language Models responds to Contextcited by 1
  8. 2025MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelismcited by 1
  9. 2025Atmos-Bench: 3D Atmospheric Structures for Climate Insightcited by 0
  10. 2026TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Adaptive Gating in Mixture-of-Experts based Language Models
  2. 2026Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
  3. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specialization
  4. 2025TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training
  5. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference
  6. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  7. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  8. 2025BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference