# Reconciling Kaplan and Chinchilla Scaling Laws

arXiv: https://arxiv.org/abs/2406.12907 · HTML: https://mlsanity.com/paper/2406.12907/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Unified Scaling Laws for Routed Language Models](https://mlsanity.com/paper/2202.01169/) (2022)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations](https://mlsanity.com/paper/2405.18392/) (2024)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [Scaling Laws for Neural Machine Translation](https://mlsanity.com/paper/2109.07740/) (2021)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)
- [Transformers Can Do Arithmetic with the Right Embeddings](https://mlsanity.com/paper/2405.17399/) (2024)
- [Chinchilla Scaling: A replication attempt](https://mlsanity.com/paper/2404.10102/) (2024)
- [D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models](https://mlsanity.com/paper/2406.01375/) (2024)
- [Scaling Laws for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2302.09650/) (2023)
- [Unraveling the Mystery of Scaling Laws: Part I](https://mlsanity.com/paper/2403.06563/) (2024)

## Built on by (most cited first, 30 of 39)

- [Resolving Discrepancies in Compute-Optimal Scaling of Language Models](https://mlsanity.com/paper/2406.19146/) (2024)
- [How Does Critical Batch Size Scale in Pre-training?](https://mlsanity.com/paper/2410.21676/) (2024)
- [Distillation Scaling Laws](https://mlsanity.com/paper/2502.08606/) (2025)
- [Scaling Laws for Native Multimodal Models](https://mlsanity.com/paper/2504.07951/) (2025)
- [Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient](https://mlsanity.com/paper/2502.05172/) (2025)
- [Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check](https://mlsanity.com/paper/2507.00885/) (2025)
- [Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning](https://mlsanity.com/paper/2509.25300/) (2025)
- [(Mis)Fitting: A Survey of Scaling Laws](https://mlsanity.com/paper/2502.18969/) (2025)
- [Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation](https://mlsanity.com/paper/2508.13144/) (2025)
- [Scaling Laws for Pre-training Agents and World Models](https://mlsanity.com/paper/2411.04434/) (2024)
- [Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks](https://mlsanity.com/paper/2507.02119/) (2025)
- [On the origin of neural scaling laws: from random graphs to natural language](https://mlsanity.com/paper/2601.10684/) (2026)
- [Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions](https://mlsanity.com/paper/2503.03862/) (2025)
- [Parameterized Synthetic Text Generation with SimpleStories](https://mlsanity.com/paper/2504.09184/) (2025)
- [Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention](https://mlsanity.com/paper/2605.29548/) (2026)
- [On the Origin of Algorithmic Progress in AI](https://mlsanity.com/paper/2511.21622/) (2025)
- [Scaling laws for amplitude surrogates](https://mlsanity.com/paper/2601.13308/) (2026)
- [The Design Space of Tri-Modal Masked Diffusion Models](https://mlsanity.com/paper/2602.21472/) (2026)
- [Pretraining Scaling Laws for Generative Evaluations of Language Models](https://mlsanity.com/paper/2509.24012/) (2025)
- [xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity](https://mlsanity.com/paper/2510.02228/) (2025)
- [From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis](https://mlsanity.com/paper/2512.13491/) (2025)
- [Theoretical Foundations of Scaling Law in Familial Models](https://mlsanity.com/paper/2512.23407/) (2025)
- [JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation](https://mlsanity.com/paper/2602.00800/) (2026)
- [Learning Rate Transfer in Normalized Transformers](https://mlsanity.com/paper/2604.27077/) (2026)
- [Skaling: Chinchilla's Exponents Meet Kaplan's Coupling](https://mlsanity.com/paper/2608.07222/) (2026)
- [Meek Models Shall Inherit the Earth](https://mlsanity.com/paper/2507.07931/) (2025)
- [Evaluating the Robustness of Chinchilla Compute-Optimal Scaling](https://mlsanity.com/paper/2509.23963/) (2025)
- [Zero-Shot Performance Prediction for Probabilistic Scaling Laws](https://mlsanity.com/paper/2510.16743/) (2025)
- [Small-Scale Experiments: Are We There Yet?](https://mlsanity.com/paper/2608.11859/) (2026)
- [Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits](https://mlsanity.com/paper/2603.22339/) (2026)

## Abstract (arXiv)

Kaplan et al. [2020] (`Kaplan') and Hoffmann et al. [2022] (`Chinchilla') studied the scaling behavior of transformers trained on next-token language prediction. These studies produced different estimates for how the number of parameters (N) and training tokens (D) should be set to achieve the lowest possible loss for a given compute budget (C). Kaplan: Noptimal propto C0.73, Chinchilla: Noptimal propto C0.50. This paper finds that much of this discrepancy can be attributed to Kaplan counting non-embedding rather than total parameters, combined with their analysis being performed at small scale. Simulating the Chinchilla study under these conditions produces biased scaling coefficients close to Kaplan's. Hence, this paper reaffirms Chinchilla's scaling coefficients, by explaining the primary cause of Kaplan's original overestimation. As a second contribution, the paper explains differences in the reported relationships between loss and compute. These findings lead us to recommend that future scaling studies use total parameters and compute.