# Chinchilla Scaling: A replication attempt

arXiv: https://arxiv.org/abs/2404.10102 · HTML: https://mlsanity.com/paper/2404.10102/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 16 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Builds on (most cited first, 6 of 6)

- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://mlsanity.com/paper/2401.00448/) (2023)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [A Dynamical Model of Neural Scaling Laws](https://mlsanity.com/paper/2402.01092/) (2024)

## Built on by (most cited first, 30 of 72)

- [TabDPT: Scaling Tabular Foundation Models on Real Data](https://mlsanity.com/paper/2410.18164/) (2024)
- [OverThink: Slowdown Attacks on Reasoning LLMs](https://mlsanity.com/paper/2502.02542/) (2025)
- [Scaling Laws for Precision](https://mlsanity.com/paper/2411.04330/) (2024)
- [Resolving Discrepancies in Compute-Optimal Scaling of Language Models](https://mlsanity.com/paper/2406.19146/) (2024)
- [How Does Critical Batch Size Scale in Pre-training?](https://mlsanity.com/paper/2410.21676/) (2024)
- [Distillation Scaling Laws](https://mlsanity.com/paper/2502.08606/) (2025)
- [Scaling Laws in Linear Regression: Compute, Parameters, and Data](https://mlsanity.com/paper/2406.08466/) (2024)
- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training](https://mlsanity.com/paper/2505.13738/) (2025)
- [Reconciling Kaplan and Chinchilla Scaling Laws](https://mlsanity.com/paper/2406.12907/) (2024)
- [Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection](https://mlsanity.com/paper/2502.06042/) (2025)
- [The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training](https://mlsanity.com/paper/2501.18965/) (2025)
- [Superposition Yields Robust Neural Scaling](https://mlsanity.com/paper/2505.10465/) (2025)
- [A Hitchhiker's Guide to Scaling Law Estimation](https://mlsanity.com/paper/2410.11840/) (2024)
- [(Mis)Fitting: A Survey of Scaling Laws](https://mlsanity.com/paper/2502.18969/) (2025)
- [How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models](https://mlsanity.com/paper/2604.21106/) (2026)
- [Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling](https://mlsanity.com/paper/2409.15156/) (2024)
- [Sparsing Law: Towards Large Language Models with Greater Activation Sparsity](https://mlsanity.com/paper/2411.02335/) (2024)
- [Gemstones: A Model Suite for Multi-Faceted Scaling Laws](https://mlsanity.com/paper/2502.06857/) (2025)
- [Bayesian scaling laws for in-context learning](https://mlsanity.com/paper/2410.16531/) (2024)
- [Language Models Improve When Pretraining Data Matches Target Tasks](https://mlsanity.com/paper/2507.12466/) (2025)
- [On the Limitations of Compute Thresholds as a Governance Strategy](https://mlsanity.com/paper/2407.05694/) (2024)
- [MuLoCo: Muon is a practical inner optimizer for DiLoCo](https://mlsanity.com/paper/2505.23725/) (2025)
- [Loss-to-Loss Prediction: Scaling Laws for All Datasets](https://mlsanity.com/paper/2411.12925/) (2024)
- [Neural Scaling Laws Rooted in the Data Distribution](https://mlsanity.com/paper/2412.07942/) (2024)
- [On the origin of neural scaling laws: from random graphs to natural language](https://mlsanity.com/paper/2601.10684/) (2026)
- [Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding](https://mlsanity.com/paper/2406.12295/) (2024)
- [Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream](https://mlsanity.com/paper/2411.05712/) (2024)
- [Improved Scaling Laws in Linear Regression via Data Reuse](https://mlsanity.com/paper/2506.08415/) (2025)
- [Scaling with Collapse: Efficient and Predictable Training of LLM Families](https://mlsanity.com/paper/2509.25087/) (2025)

## Abstract (arXiv)

Hoffmann et al. (2022) propose three methods for estimating a compute-optimal scaling law. We attempt to replicate their third estimation procedure, which involves fitting a parametric loss function to a reconstruction of data from their plots. We find that the reported estimates are inconsistent with their first two estimation methods, fail at fitting the extracted data, and report implausibly narrow confidence intervals--intervals this narrow would require over 600,000 experiments, while they likely only ran fewer than 500. In contrast, our rederivation of the scaling law using the third approach yields results that are compatible with the findings from the first two estimation procedures described by Hoffmann et al.