Data to 7 Oct 2026

Deep Learning Scaling is Predictable, Empirically

arXiv 1712.00409 ↗First posted 1 Dec 2017cs.LG stat.MLcited by 556

Not on the map. Why?

Cited by 25 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep learning (DL) creates impactful advances following a virtuous recipe: model architecture search, creating large training data sets, and scaling computation. It is widely believed that growing training sets and models should improve accuracy and result in better products. As DL application domains grow, we would like a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements to advance the state-of-the-art. This paper presents a large scale empirical characterization of generalization error and model size growth as training sets grow. We introduce a methodology for this measurement and test four machine learning domains: machine translation, language modeling, image processing, and speech recognition. Our empirical results show power-law generalization error scaling across a breadth of factors, resulting in power-law exponents---the "steepness" of the learning curve---yet to be explained by theoretical work. Further, model improvements only shift the error but do not appear to affect the power-law exponent. We also show that model size scales sublinearly with data size. These scaling relationships have significant implications on deep learning research, practice, and systems. They can assist model debugging, setting accuracy targets, and decisions about data set growth. They can also guide computing system design and underscore the importance of continued computational scaling.

Key facts. Cited by 556 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 54; 2022: 58; 2023: 77; 2024: 116; 2025: 132; 2026: 119.

556 papers.

Sort:

By area:

2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2023GPT-4 Technical Reportcited by 18,132
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 701
2024KAN: Kolmogorov-Arnold Networkscited by 634
2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
2022Mixture-of-Experts with Expert Choice Routingcited by 611

38 more

2023Textbooks Are All You Needcited by 561
2023The Falcon Series of Open Language Modelscited by 526
2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
2022MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fieldscited by 467
2024The Platonic Representation Hypothesiscited by 362
2021Explaining Neural Scaling Lawscited by 353
2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
2021Unsolved Problems in ML Safetycited by 291
2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
2025Learning to Reason under Off-Policy Guidancecited by 266
2023Challenges and Applications of Large Language Modelscited by 251
2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
2021Accelerating Sparse Deep Neural Networkscited by 213
2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
2022Compute Trends Across Three Eras of Machine Learningcited by 161
2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 140
2023The Quantization Model of Neural Scalingcited by 137
2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
2022Revisiting Neural Scaling Laws in Language and Visioncited by 129
2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
2021Scaling Laws for Neural Machine Translationcited by 110
2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
2023Explaining grokking through circuit efficiencycited by 105
2024A Dynamical Model of Neural Scaling Lawscited by 105

50 most cited of 556 All 556 (JSON)