mlsanityData to 7 Oct 2026

Training Compute-Optimal Large Language Models

arXiv 2203.15556 ↗First posted 29 Mar 2022cs.CL cs.LGcited by 2,927

Not on the map. Why?

Cited by 297 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4× more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.

Key facts. Cited by 2,927 papers in the map (data through 7 Oct 2026). Builds on 42 of its 78 references that are in the map. First posted on arXiv 29 Mar 2022.

Builds on

42 of 78 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
30 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2018Model Cards for Model Reportingcited by 1,068
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  13. 2021Ethical and social risks of harm from Language Modelscited by 957
  14. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  15. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  16. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  17. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  18. 2014Convex Optimization: Algorithms and Complexitycited by 569
  19. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  20. 2018Gender Bias in Coreference Resolutioncited by 438
  21. 2018An Empirical Model of Large-Batch Trainingcited by 268
  22. 2021Scaling Laws for Transfercited by 265
  23. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  24. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  25. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  26. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  27. 2021Challenges in Detoxifying Language Modelscited by 169
  28. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
  29. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  30. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 166; 2023: 548; 2024: 644; 2025: 739; 2026: 830.

2,927 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Segment Anythingcited by 8,158
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  12. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  13. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  14. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  15. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  16. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  17. 2023PaLM 2 Technical Reportcited by 1,154
  18. 2022Editing Models with Task Arithmeticcited by 1,076
  19. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  20. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  21. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  22. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  23. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  24. 2025Large Language Diffusion Modelscited by 872
  25. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  26. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  27. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  28. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  29. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  30. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  31. 2022A Generalist Agentcited by 802
  32. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  33. 2023StarCoder: may the source be with you!cited by 790
  34. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  35. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  36. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  37. 2023Large Language Modelscited by 774
  38. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767

50 most cited of 2,927 All 2,927 (JSON)