mlsanityData to 14 Oct 2026

Resolving Discrepancies in Compute-Optimal Scaling of Language Models

arXiv 2406.19146 ↗First posted 27 Jun 2024cs.LG cs.CLcited by 88

Not on the map. Why?

Abstract (arXiv)

Kaplan et al. and Hoffmann et al. developed influential scaling laws for the optimal model size as a function of the compute budget, but these laws yield substantially different predictions. We explain the discrepancy by reproducing the Kaplan scaling law on two datasets (OpenWebText2 and RefinedWeb) and identifying three factors causing the difference: last layer computational cost, warmup duration, and scale-dependent optimizer tuning. With these factors corrected, we obtain excellent agreement with the Hoffmann et al. (i.e., "Chinchilla") scaling law. Counter to a hypothesis of Hoffmann et al., we find that careful learning rate decay is not essential for the validity of their scaling law. As a secondary result, we derive scaling laws for the optimal learning rate and batch size, finding that tuning the AdamW β2 parameter is essential at lower batch sizes.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Builds on 51 of its 62 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

51 of 62 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2016Layer Normalizationcited by 5,669
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
38 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  7. 2021Scaling Vision Transformerscited by 999
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  9. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  10. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  12. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  13. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  14. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  15. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  16. 2023Scaling Data-Constrained Language Modelscited by 403
  17. 2018An Empirical Model of Large-Batch Trainingcited by 281
  18. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  19. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  20. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  21. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  22. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  23. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
  24. 2022A Solvable Model of Neural Scaling Lawscited by 116
  25. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  26. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  27. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  28. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 106
  29. 2024Stable LM 2 1.6B Technical Reportcited by 90
  30. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 89
  31. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 77
  32. 2024Chinchilla Scaling: A replication attemptcited by 72
  33. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 68
  34. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
  35. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 57
  36. 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 39
  37. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  38. 2022The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimescited by 25

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 33; 2026: 44.

88 papers.

Sort:

By area:

  1. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  2. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  3. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  4. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  5. 2025Distillation Scaling Lawscited by 67
  6. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
  7. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
  8. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 41
  9. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 41
  10. 2024Establishing Task Scaling Laws via Compute-Efficient Model Ladderscited by 35
  11. 2025Scaling Laws for Forgetting during Finetuning with Pretraining Data Injectioncited by 35
  12. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 35
38 more
  1. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretrainingcited by 34
  2. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 31
  3. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 27
  4. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 26
  5. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 23
  6. 2024Scaling Laws for Pre-training Agents and World Modelscited by 23
  7. 2026Deriving Neural Scaling Laws from the statistics of natural languagecited by 23
  8. 2024Scaling Law for Language Models Training Considering Batch Sizecited by 22
  9. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 20
  10. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 18
  11. 2025Efficient Prediction of Pass@k Scaling in Large Language Modelscited by 16
  12. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  13. 2026On the origin of neural scaling laws: from random graphs to natural languagecited by 14
  14. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 13
  15. 2024Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectracited by 9
  16. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 9
  17. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 8
  18. 2025Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasetscited by 8
  19. 2025Optimal Scaling Needs Optimal Normcited by 8
  20. 2025Learning in Compact Spaces with Approximately Normalized Transformercited by 6
  21. 2025Relative Scaling Laws for LLMscited by 6
  22. 2025On the Origin of Algorithmic Progress in AIcited by 6
  23. 2026Scaling laws for amplitude surrogatescited by 6
  24. 2026Scaling Laws for Mixture Pretraining Under Data Constraintscited by 6
  25. 2025Pretraining Scaling Laws for Generative Evaluations of Language Modelscited by 5
  26. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
  27. 2025Characterization and Mitigation of Training Instabilities in Microscaling Formatscited by 4
  28. 2025Efficient Hyperparameter Tuning via Trajectory Invariance Principlecited by 4
  29. 2026Output Embedding Centering for Stable LLM Pretrainingcited by 4
  30. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  31. 2026Scale Dependent Data Duplicationcited by 4
  32. 2026Compute Optimal Tokenizationcited by 4
  33. 2026Internal Data Repetition Destroys Language Modelscited by 4
  34. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexitycited by 3
  35. 2025Theoretical Foundations of Scaling Law in Familial Modelscited by 3
  36. 2026Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Ratecited by 3
  37. 2026Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selectioncited by 3
  38. 2026Modular Pretraining Enables Access Controlcited by 3

50 most cited of 88 All 88 (JSON)