# Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection

arXiv: https://arxiv.org/abs/2604.22753 · HTML: https://mlsanity.com/paper/2604.22753/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 24 Apr 2026.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance](https://mlsanity.com/paper/2403.16952/) (2024)
- [Scaling Laws for Fine-Grained Mixture of Experts](https://mlsanity.com/paper/2402.07871/) (2024)
- [Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations](https://mlsanity.com/paper/2405.18392/) (2024)
- [Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies](https://mlsanity.com/paper/2407.13623/) (2024)
- [Broken Neural Scaling Laws](https://mlsanity.com/paper/2210.14891/) (2022)
- [Resolving Discrepancies in Compute-Optimal Scaling of Language Models](https://mlsanity.com/paper/2406.19146/) (2024)
- [D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models](https://mlsanity.com/paper/2406.01375/) (2024)
- [Selecting Large Language Model to Fine-tune via Rectified Scaling Law](https://mlsanity.com/paper/2402.02314/) (2024)
- [Parallel Scaling Law for Language Models](https://mlsanity.com/paper/2505.10475/) (2025)
- [(Mis)Fitting: A Survey of Scaling Laws](https://mlsanity.com/paper/2502.18969/) (2025)
- [Scaling Laws for Upcycling Mixture-of-Experts Language Models](https://mlsanity.com/paper/2502.03009/) (2025)
- [Can Language Models Discover Scaling Laws?](https://mlsanity.com/paper/2507.21184/) (2025)
- [Sample Complexity and Representation Ability of Test-time Scaling Paradigms](https://mlsanity.com/paper/2506.05295/) (2025)
- [On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer](https://mlsanity.com/paper/2603.09952/) (2026)
- [Goal-Oriented Bayesian Optimal Experimental Design for Nonlinear Models using Markov Chain Monte Carlo](https://mlsanity.com/paper/2403.18072/) (2024)
- [Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models](https://mlsanity.com/paper/2506.10972/) (2025)
- [How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines](https://mlsanity.com/paper/2502.12051/) (2025)
- [Designs for Generalized Linear Models](https://mlsanity.com/paper/1510.05253/) (2015)

## Built on by (most cited first, 3 of 3)

- [Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search](https://mlsanity.com/paper/2609.01431/) (2026)
- [Amortizing Scaling Law Construction Costs](https://mlsanity.com/paper/2609.05016/) (2026)
- [SLDBench v2: Active Scaling Law Discovery](https://mlsanity.com/paper/2610.16297/) (2026)

## Abstract (arXiv)

Scaling laws are used to plan multi-million-dollar training runs, but fitting those laws can itself cost millions. In modern large-scale workflows, assembling a sufficiently informative set of pilot experiments is already a major budget-allocation problem rather than a routine preprocessing step. We formulate scaling-law fitting as budget-aware sequential experimental design: given a finite pool of runnable experiments with heterogeneous costs, choose which runs to execute so as to maximize extrapolation accuracy in a high-cost target region. We propose SL2 (Scaling Laws, Spend Less), an uncertainty-aware method for sequentially allocating experimental budget toward the runs most useful for target-region extrapolation. Across a diverse benchmark of scaling-law tasks, SL2 outperforms classical design-based baselines, and often approaches the performance of fitting on the full experimental set while using only about 10% of the total training budget. Our code is available at https://github.com/PlanarG/active-sl.