mlsanityData to 13 Oct 2026

Revisiting Neural Scaling Laws in Language and Vision

arXiv 2209.06640 ↗First posted 13 Sep 2022cs.LG cs.AIcited by 130

Not on the map. Why?

Abstract (arXiv)

The remarkable progress in deep learning in recent years is largely driven by improvements in scale, where bigger models are trained on larger datasets for longer schedules. To predict the benefit of scale empirically, we argue for a more rigorous methodology based on the extrapolation loss, instead of reporting the best-fitting (interpolating) parameters. We then present a recipe for estimating scaling law parameters reliably from learning curves. We demonstrate that it extrapolates more accurately than previous methods in a wide range of architecture families across several domains, including image classification, neural machine translation (NMT) and language modeling, in addition to tasks from the BIG-Bench evaluation benchmark. Finally, we release a benchmark dataset comprising of 90 evaluation tasks to facilitate research in this domain.

Key facts. Cited by 130 papers in the map (data through 13 Oct 2026). Builds on 26 of its 50 references that are in the map. First posted on arXiv 13 Sep 2022.

Builds on

26 of 50 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  5. 2020Scaling Laws for Neural Language Modelscited by 6,494
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,426
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,985
  8. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,442
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,219
  10. 2021Scaling Vision Transformerscited by 998
  11. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 900
14 more
  1. 2019Big Transfer (BiT): General Visual Representation Learningcited by 768
  2. 2017Deep Learning Scaling is Predictable, Empiricallycited by 569
  3. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  4. 2021ResNet strikes back: An improved training procedure in timmcited by 367
  5. 2021Explaining Neural Scaling Lawscited by 355
  6. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  7. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 205
  8. 2021Scaling Laws for Neural Machine Translationcited by 111
  9. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  10. 2021Learning Curve Theorycited by 81
  11. 2020A Theory of Universal Learningcited by 60
  12. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  13. 2015How much data is needed to train a medical image deep learning system to achieve necessary high accuracy?cited by 45
  14. 2021Scaling Laws for Deep Learningcited by 25

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 4; 2023: 26; 2024: 40; 2025: 46; 2026: 14.

130 papers.

Sort:

By area:

  1. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 764
  2. 2023Scaling Data-Constrained Language Modelscited by 398
  3. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 286
  4. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 286
  5. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 182
  6. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  7. 2024Language models scale reliably with over-training and on downstream taskscited by 109
  8. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  9. 2022Broken Neural Scaling Lawscited by 105
  10. 2024Scaling Laws for Precisioncited by 94
  11. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 89
  12. 2022Training Trajectories of Language Models Across Scalescited by 81
38 more
  1. 2024Symbolic Regression with a Learned Concept Librarycited by 78
  2. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  3. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 55
  4. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  5. 2023End-to-End Temporal Action Detection with 1B Parameters Across 1000 Framescited by 47
  6. 2023Federated Conformal Predictors for Distributed Uncertainty Quantificationcited by 46
  7. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  8. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  9. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  10. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  11. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 41
  12. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 41
  13. 2024How predictable is language model benchmark performance?cited by 36
  14. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 36
  15. 2024CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?cited by 35
  16. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
  17. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 30
  18. 2024Scaling Trends for Data Poisoning in LLMscited by 29
  19. 2024Scaling laws for learning with real and surrogate datacited by 27
  20. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  21. 2024DeepGate3: Towards Scalable Circuit Representation Learningcited by 26
  22. 2024Next Patch Prediction for Autoregressive Visual Generationcited by 25
  23. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 24
  24. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 24
  25. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  26. 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
  27. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 20
  28. 2025Scaling Laws For Scalable Oversightcited by 20
  29. 2025Scaling Pre-training to One Hundred Billion Data for Vision Language Modelscited by 19
  30. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 19
  31. 2024Bayesian scaling laws for in-context learningcited by 18
  32. 2025SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoningcited by 18
  33. 2023Performance Scaling via Optimal Transport: Enabling Data Selection from Partially Revealed Sourcescited by 16
  34. 2024Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Modelcited by 16
  35. 2023ArTST: Arabic Text and Speech Transformercited by 14
  36. 2024Neural Scaling Laws Rooted in the Data Distributioncited by 14
  37. 2025Data Scaling Laws for End-to-End Autonomous Drivingcited by 14
  38. 2023The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deploymentcited by 11

50 most cited of 130 All 130 (JSON)