mlsanityData to 14 Oct 2026

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

arXiv 2112.11446 ↗First posted 8 Dec 2021cs.CL cs.AIcited by 1,202

Not on the map. Why?

Cited by 174 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales -- from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model's behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the mitigation of downstream harms.

Key facts. Cited by 1,202 papers in the map (data through 14 Oct 2026). Builds on 66 of its 189 references that are in the map. First posted on arXiv 8 Dec 2021.

Builds on

66 of 189 references.

References · 66

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2016Layer Normalizationcited by 5,669
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
38 more
  1. 2016Deep Learning with Differential Privacycited by 2,728
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2019Root Mean Square Layer Normalizationcited by 1,480
  10. 2018Datasheets for Datasetscited by 1,386
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  13. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  14. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  15. 2020What is the State of Neural Network Pruning?cited by 583
  16. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  17. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  18. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  19. 2019The State of Sparsity in Deep Neural Networkscited by 476
  20. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  21. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  22. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  23. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 404
  24. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  25. 2018AI safety via debatecited by 400
  26. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  27. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  28. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 337
  29. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  30. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  31. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  32. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  33. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  34. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 196
  35. 2016Exploring the Limits of Language Modelingcited by 173
  36. 2019MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claimscited by 169
  37. 2021Alignment of Language Agentscited by 168
  38. 2018Supervising strong learners by amplifying weak expertscited by 145

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 232; 2023: 403; 2024: 305; 2025: 178; 2026: 79.

1,202 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
38 more
  1. 2023Qwen Technical Reportcited by 3,344
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  13. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  14. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  15. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  16. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  17. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  18. 2022Red Teaming Language Models with Language Modelscited by 1,026
  19. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  20. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  21. 2023BloombergGPT: A Large Language Model for Financecited by 939
  22. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  23. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  24. 2022A Generalist Agentcited by 808
  25. 2023StarCoder: may the source be with you!cited by 799
  26. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  27. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  28. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  29. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  30. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  31. 2024Yi: Open Foundation Models by 01.AIcited by 719
  32. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  33. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  34. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  35. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  36. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  37. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  38. 2023A Comprehensive Overview of Large Language Modelscited by 648

50 most cited of 1,202 All 1,202 (JSON)