mlsanityData to 7 Oct 2026

What Language Model to Train if You Have One Million GPU Hours?

arXiv 2210.15424 ↗First posted 27 Oct 2022cs.CL cs.AI cs.LGcited by 92

Not on the map. Why?

Abstract (arXiv)

The crystallization of modeling methods around the Transformer architecture has been a boon for practitioners. Simple, well-motivated architectural variations can transfer across tasks and scale, increasing the impact of modeling research. However, with the emergence of state-of-the-art 100B+ parameters models, large language models are increasingly expensive to accurately design and train. Notably, it can be difficult to evaluate how modeling decisions may impact emergent capabilities, given that these capabilities arise mainly from sheer scale alone. In the process of building BLOOM--the Big Science Large Open-science Open-access Multilingual language model--our goal is to identify an architecture and training setup that makes the best use of our 1,000,000 A100-GPU-hours budget. Specifically, we perform an ablation study at the billion-parameter scale comparing different modeling practices and their impact on zero-shot generalization. In addition, we study the impact of various popular pre-training corpora on zero-shot generalization. We also study the performance of a multilingual model and how it compares to the English-only one. Finally, we consider the scaling behaviour of Transformers to choose the target model size, shape, and training setup. All our models and code are open-sourced at https://huggingface.co/bigscience .

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 56 of its 88 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

56 of 88 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2016Layer Normalizationcited by 5,579
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2020GLU Variants Improve Transformercited by 1,724
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  13. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  14. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  15. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  16. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  17. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  18. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  19. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  20. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  21. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  22. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  23. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  24. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  25. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  26. 2016Language Modeling with Gated Convolutional Networkscited by 688
  27. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  28. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  29. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  30. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  31. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  32. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  33. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  34. 2016Using the Output Embedding to Improve Language Modelscited by 266
  35. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  36. 2022Predictability and Surprise in Large Generative Modelscited by 209
  37. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  38. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 33; 2024: 30; 2025: 17; 2026: 5.

92 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2023BloombergGPT: A Large Language Model for Financecited by 923
  5. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  6. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  7. 2023The Falcon Series of Open Language Modelscited by 526
  8. 2023Scaling Data-Constrained Language Modelscited by 391
  9. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  10. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  11. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  12. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
38 more
  1. 2024Generative Representational Instruction Tuningcited by 227
  2. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  3. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  4. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  5. 2023Large Language Models in Law: A Surveycited by 128
  6. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  7. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  8. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  9. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
  10. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  11. 2024EuroLLM: Multilingual Language Models for Europecited by 103
  12. 2024Scaling Laws for Precisioncited by 92
  13. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  14. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  15. 2022Training Trajectories of Language Models Across Scalescited by 80
  16. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  17. 2023Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Taskscited by 59
  18. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  19. 2023FinGPT: Large Generative Models for a Small Languagecited by 49
  20. 2024xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Proteincited by 49
  21. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 45
  22. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuningcited by 39
  23. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  24. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 39
  25. 2023Self-Influence Guided Data Reweighting for Language Model Pre-trainingcited by 37
  26. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  27. 2024PALO: A Polyglot Large Multimodal Model for 5B Peoplecited by 34
  28. 2024A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelismcited by 33
  29. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 27
  30. 2023Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learningcited by 26
  31. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  32. 2023Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddingscited by 23
  33. 2024Breaking the Length Barrier: LLM-Enhanced CTR Prediction in Long Textual User Behaviorscited by 23
  34. 2024Causal Estimation of Memorisation Profilescited by 20
  35. 2023BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Modelcited by 19
  36. 2023Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Promptscited by 19
  37. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 19
  38. 2024Deep Correlated Prompting for Visual Recognition with Missing Modalitiescited by 17

50 most cited of 92 All 92 (JSON)