# Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning

arXiv: https://arxiv.org/abs/2605.17234 · HTML: https://mlsanity.com/paper/2605.17234/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 67 references that are in the map.
- First posted on arXiv 17 May 2026.

## Builds on (most cited first, 30 of 39)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice](https://mlsanity.com/paper/2007.15745/) (2020)
- [Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations](https://mlsanity.com/paper/2405.18392/) (2024)
- [GPflow: A Gaussian process library using TensorFlow](https://mlsanity.com/paper/1610.08733/) (2016)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [Revisiting Neural Scaling Laws in Language and Vision](https://mlsanity.com/paper/2209.06640/) (2022)
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [Scaling Laws for Neural Machine Translation](https://mlsanity.com/paper/2109.07740/) (2021)
- [Exploring the Limits of Large Scale Pre-training](https://mlsanity.com/paper/2110.02095/) (2021)
- [What Language Model to Train if You Have One Million GPU Hours?](https://mlsanity.com/paper/2210.15424/) (2022)
- [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)
- [Chinchilla Scaling: A replication attempt](https://mlsanity.com/paper/2404.10102/) (2024)
- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [Data Scaling Laws in NMT: The Effect of Noise and Architecture](https://mlsanity.com/paper/2202.01994/) (2022)
- [Predicting Performance for Natural Language Processing Tasks](https://mlsanity.com/paper/2005.00870/) (2020)
- [Scaling Laws for Predicting Downstream Performance in LLMs](https://mlsanity.com/paper/2410.08527/) (2024)
- [Scaling Optimal LR Across Token Horizons](https://mlsanity.com/paper/2409.19913/) (2024)
- [Reconciling Kaplan and Chinchilla Scaling Laws](https://mlsanity.com/paper/2406.12907/) (2024)
- [Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks](https://mlsanity.com/paper/2310.20447/) (2023)

## Abstract (arXiv)

Predicting model performance at larger scales enables the design of training strategies and architectures tailored to specific performance targets. Empirical scaling law research identifies functional forms to aid this prediction task. These describe the relationship between loss and compute using a loss-compute frontier defined by learning curves. Due to the empirical nature of this approach, the computational burden is substantial, making strategic resource allocation essential - yet it remains surprisingly underexplored. In this work, we address this shortcoming by exploring the suitability of Successive Halving (SH) and SH combined with parametric and non-parametric surrogate models. In addition to enabling a more systematic allocation of a given compute budget, our findings show that SH paired with surrogate models yields a set of learning curves that includes one with a lower loss-compute value than what naive uniform allocation or an SH-only approach can obtain. Our experiments demonstrate mean relative improvements of up to 2.84% and 5.47% on real-world and synthetic learning curve datasets. This strategic resource allocation enables us to obtain accurate scaling laws at significantly reduced computational costs, saving up to 98.7% over the traditional exhaustive approach.