mlsanityData to 7 Oct 2026

Scaling Laws for Neural Language Models

arXiv 2001.08361 ↗First posted 23 Jan 2020cs.LG stat.MLcited by 6,411

Not on the map. Why?

Cited by 591 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.

Key facts. Cited by 6,411 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 3; 2021: 209; 2022: 321; 2023: 789; 2024: 1503; 2025: 2002; 2026: 1584.

6,411 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2023Segment Anythingcited by 8,158
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  10. 2022Scalable Diffusion Models with Transformerscited by 5,931
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
38 more
  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
  13. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  14. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  15. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  16. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  17. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  18. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  19. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  20. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  21. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  22. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  23. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  24. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  25. 2025s1: Simple test-time scalingcited by 1,311
  26. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  27. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  28. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  29. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  30. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  31. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  32. 2023PaLM 2 Technical Reportcited by 1,154
  33. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  34. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  35. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  36. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  37. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  38. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020

50 most cited of 6,411 All 6,411 (JSON)