# Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits

arXiv: https://arxiv.org/abs/2603.22339 · HTML: https://mlsanity.com/paper/2603.22339/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 48 references that are in the map.
- First posted on arXiv 21 Mar 2026.

## Builds on (most cited first, 30 of 39)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://mlsanity.com/paper/2401.00448/) (2023)
- [Scaling Laws for Precision](https://mlsanity.com/paper/2411.04330/) (2024)
- [Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design](https://mlsanity.com/paper/2305.13035/) (2023)
- [Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic](https://mlsanity.com/paper/2404.07177/) (2024)
- [Chinchilla Scaling: A replication attempt](https://mlsanity.com/paper/2404.10102/) (2024)
- [Scaling Laws for Optimal Data Mixtures](https://mlsanity.com/paper/2507.09404/) (2025)
- [Scaling Laws for Native Multimodal Models](https://mlsanity.com/paper/2504.07951/) (2025)
- [Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models](https://mlsanity.com/paper/2507.17702/) (2025)
- [Scaling Laws For Diffusion Transformers](https://mlsanity.com/paper/2410.08184/) (2024)
- [Reconciling Kaplan and Chinchilla Scaling Laws](https://mlsanity.com/paper/2406.12907/) (2024)
- [Beyond Language Modeling: An Exploration of Multimodal Pretraining](https://mlsanity.com/paper/2603.03276/) (2026)
- [Establishing Task Scaling Laws via Compute-Efficient Model Ladders](https://mlsanity.com/paper/2412.04403/) (2024)
- [Scaling Behavior of Discrete Diffusion Language Models](https://mlsanity.com/paper/2512.10858/) (2025)
- [MuPT: A Generative Symbolic Music Pretrained Transformer](https://mlsanity.com/paper/2404.06393/) (2024)
- [Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data](https://mlsanity.com/paper/2103.03399/) (2021)
- [Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient](https://mlsanity.com/paper/2502.05172/) (2025)
- [Scaling Laws For Dense Retrieval](https://mlsanity.com/paper/2403.18684/) (2024)
- [(Mis)Fitting: A Survey of Scaling Laws](https://mlsanity.com/paper/2502.18969/) (2025)
- [ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality](https://mlsanity.com/paper/2510.22037/) (2025)
- [Scaling Laws of Motion Forecasting and Planning -- Technical Report](https://mlsanity.com/paper/2506.08228/) (2025)
- [Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time](https://mlsanity.com/paper/2510.01098/) (2025)
- [Can Language Models Discover Scaling Laws?](https://mlsanity.com/paper/2507.21184/) (2025)
- [Training Compute-Optimal Protein Language Models](https://mlsanity.com/paper/2411.02142/) (2024)
- [Exploring Scaling Laws for EHR Foundation Models](https://mlsanity.com/paper/2505.22964/) (2025)

## Abstract (arXiv)

Chinchilla Approach 2 is among the most widely used methods for fitting neural scaling laws. Its parabolic approximation introduces systematic biases in compute-optimal allocation estimates, even on noise-free synthetic data. Applied to published Llama 3 IsoFLOP data at open frontier compute scales, these biases imply a parameter underallocation corresponding to 6.5% of the 3.8×1025 FLOP training budget and 1.4M (90% CI: 412K-2.9M) in unnecessary compute at 50% H100 MFU. Simulated multimodal model misallocations show even greater opportunity costs due to higher loss surface asymmetry. Three sources of this error are examined: IsoFLOP sampling grid width (Taylor approximation accuracy), uncentered IsoFLOP sampling, and loss surface asymmetry (α neq β). Chinchilla Approach 3 largely eliminates these biases but is often regarded as less data-efficient, numerically unstable, prone to local minima, and harder to implement. Each concern is shown to be unfounded or addressable, especially when the partially linear structure of the objective is exploited via Variable Projection, enabling unbiased inference on all five loss surface parameters through a two-dimensional optimization that is well-conditioned, analytically differentiable, and amenable to dense, or even exhaustive, grid search. It may serve as a more convenient replacement for Approach 2 or a more scalable alternative for adaptations of Approach 3 to richer scaling law formulations. See https://github.com/Open-Athena/vpnls for details and https://openathena.ai/scaling-law-analysis for other results from this study.