mlsanityData to 7 Oct 2026

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

arXiv 2112.06905 ↗First posted 13 Dec 2021cs.CLcited by 890

On the map: World models → World models for language agents

also in 1 more
Abstract (arXiv)

Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.

Key facts. Cited by 890 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 73 of its 108 references that are in the map. First posted on arXiv 13 Dec 2021.

Builds on

73 of 108 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Language Models are Few-Shot Learners28 May

Other references · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
38 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  13. 2020GLU Variants Improve Transformercited by 1,724
  14. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  15. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  16. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  17. 2019Root Mean Square Layer Normalizationcited by 1,425
  18. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  19. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  20. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  21. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  22. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  23. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  24. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  25. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  26. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  27. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  28. 2016Language Modeling with Gated Convolutional Networkscited by 688
  29. 2014Distributed Representations of Sentences and Documentscited by 677
  30. 2018CoQA: A Conversational Question Answering Challengecited by 674
  31. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  32. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  33. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  34. 2018Gender Bias in Coreference Resolutioncited by 438
  35. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  36. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  37. 2018QuAC : Question Answering in Contextcited by 379
  38. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 88; 2023: 188; 2024: 224; 2025: 242; 2026: 146.

890 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  12. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
38 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  5. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  6. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  7. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  8. 2023A Comprehensive Overview of Large Language Modelscited by 639
  9. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  10. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  12. 2024Large Language Models: A Surveycited by 506
  13. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  14. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  15. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  16. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  17. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  18. 2023Scaling Data-Constrained Language Modelscited by 391
  19. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  20. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  21. 2023Language Models can Solve Computer Taskscited by 371
  22. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  23. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  24. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  25. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  26. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  27. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  28. 2022Confident Adaptive Language Modelingcited by 297
  29. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  30. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  31. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  32. 2022UL2: Unifying Language Learning Paradigmscited by 292
  33. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  34. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  35. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  36. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  37. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  38. 2022Efficient Training of Language Models to Fill in the Middlecited by 248

50 most cited of 890 All 890 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling Laws for Fine-Grained Mixture of Experts
  2. 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models
  3. 2026FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Models
  4. 2024A Survey on Mixture of Experts in Large Language Models
  5. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
  6. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  7. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling
  8. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference