mlsanityData to 7 Oct 2026

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

arXiv 1701.06538 ↗First posted 23 Jan 2017cs.LG cs.CL cs.NEcited by 3,413

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.

Key facts. Cited by 3,413 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 26 of its 45 references that are in the map. First posted on arXiv 23 Jan 2017.

Builds on

26 of 45 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  8. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  9. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  10. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  11. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  12. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
14 more
  1. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  2. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  3. 2014Recurrent Neural Network Regularizationcited by 189
  4. 2016Exploring the Limits of Language Modelingcited by 172
  5. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  6. 2015Distributed Gaussian Processescited by 84
  7. 2016Memory-Efficient Backpropagation Through Timecited by 78
  8. 2014Addressing the Rare Word Problem in Neural Machine Translationcited by 50
  9. 2013Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networkscited by 44
  10. 2015Generative Image Modeling Using Spatial LSTMscited by 35
  11. 2014Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learningcited by 24
  12. 2015Dynamic Capacity Networkscited by 16
  13. 2016Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translationcited by 8
  14. 2014Deep Sequential Neural Networkcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 7; 2019: 5; 2020: 8; 2021: 119; 2022: 186; 2023: 255; 2024: 608; 2025: 954; 2026: 1270.

3,413 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2024Mixtral of Expertscited by 1,785
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  9. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  11. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
38 more
  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  2. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  3. 2024A Survey on Large Language Models for Code Generationcited by 646
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  6. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  8. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  9. 2024Scaling and evaluating sparse autoencoderscited by 530
  10. 2022Efficiently Scaling Transformer Inferencecited by 529
  11. 2024Large Language Models: A Surveycited by 506
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  13. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  14. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  15. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  16. 2021Dynamic Neural Networks: A Surveycited by 405
  17. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  18. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  19. 2021A Survey of Transformerscited by 399
  20. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  21. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  22. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  23. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  24. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  25. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  26. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  27. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  28. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  29. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  30. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 280
  31. 2023Challenges and Applications of Large Language Modelscited by 251
  32. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  33. 2023From Sparse to Soft Mixtures of Expertscited by 245
  34. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  35. 2021Hash Layers For Large Sparse Modelscited by 236
  36. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  37. 2024Capabilities of Gemini Models in Medicinecited by 231
  38. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228

50 most cited of 3,413 All 3,413 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Adaptive Gating in Mixture-of-Experts based Language Models
  2. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models
  3. 2025DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
  4. 2025DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstruction
  5. 2026DOT-MoE: Differentiable Optimal Transport for MoEfication
  6. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?
  7. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
  8. 2025BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference