# Amortizing Scaling Law Construction Costs

arXiv: https://arxiv.org/abs/2609.05016 · HTML: https://mlsanity.com/paper/2609.05016/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 4 Sep 2026.

## Builds on (most cited first, 25 of 25)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Unified Scaling Laws for Routed Language Models](https://mlsanity.com/paper/2202.01169/) (2022)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design](https://mlsanity.com/paper/2305.13035/) (2023)
- [Resolving Discrepancies in Compute-Optimal Scaling of Language Models](https://mlsanity.com/paper/2406.19146/) (2024)
- [Multi-Information Source Optimization](https://mlsanity.com/paper/1603.00389/) (2016)
- [Scaling Exponents Across Parameterizations and Optimizers](https://mlsanity.com/paper/2407.05872/) (2024)
- [How Does Critical Batch Size Scale in Pre-training?](https://mlsanity.com/paper/2410.21676/) (2024)
- [Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training](https://mlsanity.com/paper/2505.13738/) (2025)
- [Scaling Optimal LR Across Token Horizons](https://mlsanity.com/paper/2409.19913/) (2024)
- [Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks](https://mlsanity.com/paper/2310.20447/) (2023)
- [Thinking inside the box: A tutorial on grey-box Bayesian optimization](https://mlsanity.com/paper/2201.00272/) (2022)
- [In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization](https://mlsanity.com/paper/2404.16795/) (2024)
- [Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling](https://mlsanity.com/paper/2405.14578/) (2024)
- [Towards Precise Scaling Laws for Video Diffusion Transformers](https://mlsanity.com/paper/2411.17470/) (2024)
- [Warmstarting for Scaling Language Models](https://mlsanity.com/paper/2411.07340/) (2024)
- [Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection](https://mlsanity.com/paper/2604.22753/) (2026)
- [Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization](https://mlsanity.com/paper/2504.10735/) (2025)
- [Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss](https://mlsanity.com/paper/2608.28308/) (2026)
- [Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search](https://mlsanity.com/paper/2609.01431/) (2026)

## Abstract (arXiv)

Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to 10--100×.