mlsanityData to 7 Oct 2026

Reconciling Kaplan and Chinchilla Scaling Laws

arXiv 2406.12907 ↗First posted 12 Jun 2024cs.LG cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Kaplan et al. [2020] (`Kaplan') and Hoffmann et al. [2022] (`Chinchilla') studied the scaling behavior of transformers trained on next-token language prediction. These studies produced different estimates for how the number of parameters (N) and training tokens (D) should be set to achieve the lowest possible loss for a given compute budget (C). Kaplan: Noptimal propto C0.73, Chinchilla: Noptimal propto C0.50. This paper finds that much of this discrepancy can be attributed to Kaplan counting non-embedding rather than total parameters, combined with their analysis being performed at small scale. Simulating the Chinchilla study under these conditions produces biased scaling coefficients close to Kaplan's. Hence, this paper reaffirms Chinchilla's scaling coefficients, by explaining the primary cause of Kaplan's original overestimation. As a second contribution, the paper explains differences in the reported relationships between loss and compute. These findings lead us to recommend that future scaling studies use total parameters and compute.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 20 of its 26 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

20 of 26 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  6. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  7. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  8. 2021Explaining Neural Scaling Lawscited by 353
  9. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  10. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  11. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  12. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
8 more
  1. 2021Scaling Laws for Neural Machine Translationcited by 110
  2. 2022A Solvable Model of Neural Scaling Lawscited by 101
  3. 2021Learning Curve Theorycited by 79
  4. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  5. 2024Chinchilla Scaling: A replication attemptcited by 65
  6. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  7. 2023Scaling Laws for Multilingual Neural Machine Translationcited by 27
  8. 2024Unraveling the Mystery of Scaling Laws: Part Icited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 18; 2026: 15.

36 papers.

Sort:

By area:

  1. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  2. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  3. 2025Distillation Scaling Lawscited by 59
  4. 2025Scaling Laws for Native Multimodal Modelscited by 47
  5. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 29
  6. 2025Scaling Laws Are Unreliable for Downstream Tasks: A Reality Checkcited by 27
  7. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 24
  8. 2024Scaling Laws for Pre-training Agents and World Modelscited by 23
  9. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoningcited by 23
  10. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 22
  11. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 21
  12. 2026On the origin of neural scaling laws: from random graphs to natural languagecited by 12
24 more
  1. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 11
  2. 2025Parameterized Synthetic Text Generation with SimpleStoriescited by 9
  3. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 8
  4. 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
  5. 2025On the Origin of Algorithmic Progress in AIcited by 5
  6. 2026Scaling laws for amplitude surrogatescited by 5
  7. 2025Pretraining Scaling Laws for Generative Evaluations of Language Modelscited by 4
  8. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexitycited by 3
  9. 2025Theoretical Foundations of Scaling Law in Familial Modelscited by 3
  10. 2025Meek Models Shall Inherit the Earthcited by 2
  11. 2025Evaluating the Robustness of Chinchilla Compute-Optimal Scalingcited by 2
  12. 2025Zero-Shot Performance Prediction for Probabilistic Scaling Lawscited by 2
  13. 2025From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesiscited by 2
  14. 2026JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulationcited by 2
  15. 2026Small-Scale Experiments: Are We There Yet?cited by 2
  16. 2026Learning Rate Transfer in Normalized Transformerscited by 1
  17. 2026Skaling: Chinchilla's Exponents Meet Kaplan's Couplingcited by 1
  18. 2026Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fitscited by 0
  19. 2026Tempus: A Temporally Scalable Resource-Invariant GEMM Streaming Framework for Versal AI Edgecited by 0
  20. 2026Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained Worldcited by 0
  21. 2026Position: Agentic AI System Is a Foreseeable Pathway to AGIcited by 0
  22. 2026Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruningcited by 0
  23. 2026How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Textcited by 0
  24. 2026RoboJEPA: Scaling Robotic Latent World Modelscited by 0