mlsanityData to 13 Oct 2026

PaLM: Scaling Language Modeling with Pathways

arXiv 2204.02311 ↗First posted 5 Apr 2022cs.CLcited by 5,317

Not on the map. Why?

Cited by 553 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies.

Key facts. Cited by 5,317 papers in the map (data through 13 Oct 2026). Builds on 127 of its 173 references that are in the map. First posted on arXiv 5 Apr 2022.

Builds on

127 of 173 references.

References · 127

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2020Language Models are Few-Shot Learnerscited by 32,320
  5. 2022Training language models to follow instructions with human feedbackcited by 15,856
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  11. 2020Scaling Laws for Neural Language Modelscited by 6,494
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
38 more
  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,657
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,628
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,999
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  7. 2021Program Synthesis with Large Language Modelscited by 3,573
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,972
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,925
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  13. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,133
  14. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,131
  15. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,979
  16. 2020Shortcut Learning in Deep Neural Networkscited by 1,944
  17. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,815
  18. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,805
  19. 2020GLU Variants Improve Transformercited by 1,759
  20. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,725
  21. 2020Big Bird: Transformers for Longer Sequencescited by 1,681
  22. 2020Reformer: The Efficient Transformercited by 1,662
  23. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,618
  24. 2020Rethinking Attention with Performerscited by 1,594
  25. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,546
  26. 2019Generating Long Sequences with Sparse Transformerscited by 1,538
  27. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,468
  28. 2022Competition-Level Code Generation with AlphaCodecited by 1,445
  29. 2018Datasheets for Datasetscited by 1,377
  30. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,312
  31. 2021Improving language models by retrieving from trillions of tokenscited by 1,286
  32. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,249
  33. 2022LaMDA: Language Models for Dialog Applicationscited by 1,219
  34. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,201
  35. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  36. 2018Model Cards for Model Reportingcited by 1,076
  37. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 973
  38. 2021Ethical and social risks of harm from Language Modelscited by 964

50 most cited of 127 All 127 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 430; 2023: 1936; 2024: 1674; 2025: 915; 2026: 362.

5,317 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,502
  4. 2023Segment Anythingcited by 8,195
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,857
  7. 2023Visual Instruction Tuningcited by 7,827
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,339
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,348
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,117
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
38 more
  1. 2023Qwen Technical Reportcited by 3,310
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,181
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,007
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,729
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,703
  6. 2023A Survey of Large Language Modelscited by 2,559
  7. 2022Emergent Abilities of Large Language Modelscited by 2,499
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,135
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,130
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,057
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,052
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,978
  13. 2022Large Language Models Encode Clinical Knowledgecited by 1,817
  14. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,785
  15. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,777
  16. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,761
  17. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,594
  18. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,465
  19. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,404
  20. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,365
  21. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,365
  22. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,323
  23. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,313
  24. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,289
  25. 2023AgentBench: Evaluating LLMs as Agentscited by 1,242
  26. 2023A Survey on Evaluation of Large Language Modelscited by 1,236
  27. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,230
  28. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,191
  29. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,180
  30. 2023PaLM 2 Technical Reportcited by 1,162
  31. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,126
  32. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,117
  33. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,112
  34. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,112
  35. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,111
  36. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,092
  37. 2023LIMA: Less Is More for Alignmentcited by 1,091
  38. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,082

50 most cited of 5,317 All 5,317 (JSON)