mlsanityData to 13 Oct 2026

Efficient Large Scale Language Modeling with Mixtures of Experts

arXiv 2112.10684 ↗First posted 20 Dec 2021cs.CL cs.AI cs.LGcited by 175

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning. With the exception of fine-tuning, we find MoEs to be substantially more compute efficient. At more modest training budgets, MoEs can match the performance of dense models using ~4 times less compute. This gap narrows at scale, but our largest MoE model (1.1T parameters) consistently outperforms a compute-equivalent dense model (6.7B parameters). Overall, this performance gap varies greatly across tasks and domains, suggesting that MoE and dense models generalize differently in ways that are worthy of future study. We make our code and models publicly available for research use.

Key facts. Cited by 175 papers in the map (data through 13 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 65 of its 93 references that are in the map. First posted on arXiv 20 Dec 2021.

Builds on

65 of 93 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 62

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2020Language Models are Few-Shot Learnerscited by 32,320
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,213
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,472
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,779
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,676
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,379
  8. 2018Deep contextualized word representationscited by 2,286
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,163
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,133
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,056
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  13. 2019Generating Long Sequences with Sparse Transformerscited by 1,538
  14. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,487
  15. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,248
  16. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,201
  17. 2017Mixed Precision Trainingcited by 1,147
  18. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,042
  19. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 935
  20. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 918
  21. 2016Training Deep Nets with Sublinear Memory Costcited by 840
  22. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 705
  23. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  24. 2019Green AIcited by 623
  25. 2021Carbon Emissions and Large Neural Network Trainingcited by 565
  26. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  27. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  28. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  29. 2020Jukebox: A Generative Model for Musiccited by 537
  30. 2019Quantifying the Carbon Emissions of Machine Learningcited by 519
  31. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 414
  32. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  33. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  34. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  35. 2021Hash Layers For Large Sparse Modelscited by 237
  36. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  37. 2018A Simple Method for Commonsense Reasoningcited by 223
  38. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 48; 2023: 40; 2024: 46; 2025: 29; 2026: 12.

175 papers.

Sort:

By area:

  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,972
  3. 2022Emergent Abilities of Large Language Modelscited by 2,499
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,424
  5. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 926
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 896
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 853
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 841
  9. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 760
  10. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  11. 2023The Falcon Series of Open Language Modelscited by 533
  12. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 497
38 more
  1. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 437
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 402
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 396
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 298
  5. 2022Unified Scaling Laws for Routed Language Modelscited by 252
  6. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  7. 2024A Survey on Mixture of Experts in Large Language Modelscited by 235
  8. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 219
  9. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 208
  10. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 207
  11. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 189
  12. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 189
  13. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 186
  14. 2023Efficient Large Language Models: A Surveycited by 185
  15. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  16. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  17. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 147
  18. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  19. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  20. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 117
  21. 2022Protein structure generation via folding diffusioncited by 114
  22. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
  23. 2024Small Language Models: Survey, Measurements, and Insightscited by 110
  24. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  25. 2024Language models scale reliably with over-training and on downstream taskscited by 109
  26. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 106
  27. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  28. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 96
  29. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  30. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  31. 2023Modular Deep Learningcited by 83
  32. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  33. 2022Training Trajectories of Language Models Across Scalescited by 81
  34. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  35. 2023LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Modelscited by 71
  36. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 70
  37. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  38. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65

50 most cited of 175 All 175 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
  2. 2024Scaling Laws for Fine-Grained Mixture of Experts
  3. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  4. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  5. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  6. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  7. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
  8. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models