mlsanityData to 7 Oct 2026

Chinchilla Scaling: A replication attempt

arXiv 2404.10102 ↗First posted 15 Apr 2024cs.AI cs.CLcited by 65

Not on the map. Why?

Abstract (arXiv)

Hoffmann et al. (2022) propose three methods for estimating a compute-optimal scaling law. We attempt to replicate their third estimation procedure, which involves fitting a parametric loss function to a reconstruction of data from their plots. We find that the reported estimates are inconsistent with their first two estimation methods, fail at fitting the extracted data, and report implausibly narrow confidence intervals--intervals this narrow would require over 600,000 experiments, while they likely only ran fewer than 500. In contrast, our rederivation of the scaling law using the third approach yields results that are compatible with the findings from the first two estimation procedures described by Hoffmann et al.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 6 of its 16 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

6 of 16 references.

References · 6

  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  3. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  4. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  5. 2023The Quantization Model of Neural Scalingcited by 137
  6. 2024A Dynamical Model of Neural Scaling Lawscited by 105

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 24; 2026: 24.

65 papers.

Sort:

By area:

  1. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 97
  2. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 94
  3. 2024Scaling Laws for Precisioncited by 92
  4. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  5. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  6. 2025Distillation Scaling Lawscited by 59
  7. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  8. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  9. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  10. 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 36
  11. 2025Scaling Laws for Forgetting during Finetuning with Pretraining Data Injectioncited by 35
  12. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 31
38 more
  1. 2025Superposition Yields Robust Neural Scalingcited by 30
  2. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 24
  3. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 23
  4. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  5. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 21
  6. 2026How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Modelscited by 20
  7. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 19
  8. 2024Bayesian scaling laws for in-context learningcited by 18
  9. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 18
  10. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
  11. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 16
  12. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  13. 2024Neural Scaling Laws Rooted in the Data Distributioncited by 14
  14. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decodingcited by 12
  15. 2024Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Streamcited by 12
  16. 2026On the origin of neural scaling laws: from random graphs to natural languagecited by 12
  17. 2025Physics of Skill Learningcited by 8
  18. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 8
  19. 2026Inverse Depth Scaling From Most Layers Being Similarcited by 8
  20. 2025Improved Scaling Laws in Linear Regression via Data Reusecited by 7
  21. 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
  22. 2025Forecasting Frontier Language Model Agent Capabilitiescited by 5
  23. 2025Compute-Optimal LLMs Provably Generalize Better With Scalecited by 5
  24. 2025Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Modelscited by 5
  25. 2025A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Searchcited by 5
  26. 2025On the Origin of Algorithmic Progress in AIcited by 5
  27. 2024Time Matters: Scaling Laws for Any Budgetcited by 4
  28. 2025DIVER-1: Scaling Intracranial EEG Foundation Models for Transferable Representationscited by 4
  29. 2026Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficientscited by 4
  30. 2025Note on Selection Bias in Observational Estimates of Algorithmic Progresscited by 3
  31. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexitycited by 3
  32. 2026Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Ratecited by 3
  33. 2026Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokenscited by 3
  34. 2026Internal Data Repetition Destroys Language Modelscited by 3
  35. 2025Reusing Overtrained Language Models Saturates Scalingcited by 2
  36. 2025Large Language Model Scaling Laws for Neural Quantum States in Quantum Chemistrycited by 1
  37. 2026How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Sizecited by 1
  38. 2026Bridging Compute- and Data-Optimal Pretrainingcited by 1

50 most cited of 65 All 65 (JSON)