mlsanityData to 7 Oct 2026

Scaling Laws for Hyperparameter Optimization

arXiv 2302.00441 ↗First posted 1 Feb 2023cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Hyperparameter optimization is an important subfield of machine learning that focuses on tuning the hyperparameters of a chosen algorithm to achieve peak performance. Recently, there has been a stream of methods that tackle the issue of hyperparameter optimization, however, most of the methods do not exploit the dominant power law nature of learning curves for Bayesian optimization. In this work, we propose Deep Power Laws (DPL), an ensemble of neural network models conditioned to yield predictions that follow a power-law scaling pattern. Our method dynamically decides which configurations to pause and train incrementally by making use of gray-box evaluations. We compare our method against 7 state-of-the-art competitors on 3 benchmarks related to tabular, image, and NLP datasets covering 59 diverse tasks. Our method achieves the best results across all benchmarks by obtaining the best any-time results compared to all competitors.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

29 of 46 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  5. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  6. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  7. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  8. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  9. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  10. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  11. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  12. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 232
17 more
  1. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 229
  2. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 219
  3. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  4. 2017Multi-fidelity Bayesian Optimisation with Continuous Approximationscited by 120
  5. 2021Scaling Laws for Neural Machine Translationcited by 110
  6. 2019An Open Source AutoML Benchmarkcited by 85
  7. 2020Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDLcited by 67
  8. 2019Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonflycited by 66
  9. 2020Provably Efficient Online Hyperparameter Optimization with Population-Based Banditscited by 61
  10. 2020Multi-Fidelity Bayesian Optimization via Deep Neural Networkscited by 46
  11. 2020On the Predictability of Pruning Across Scalescited by 33
  12. 2020Using a thousand optimization tasks to learn hyperparameter search strategiescited by 29
  13. 2022Learning Curves for Decision Making in Supervised Machine Learning: A Surveycited by 27
  14. 2020Learning to Rank Learning Curvescited by 16
  15. 2022Supervising the Multi-Fidelity Race of Hyperparameter Configurationscited by 15
  16. 2022Optimizing Data Collection for Machine Learningcited by 15
  17. 2022Pre-training helps Bayesian optimization toocited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 6; 2026: 5.

16 papers.

Sort:

By area:

  1. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  2. 2023Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and Howcited by 17
  3. 2024Position: A Call to Action for a Human-Centered AutoML Paradigmcited by 14
  4. 2024Scalable Training of Trustworthy and Energy-Efficient Predictive Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNNcited by 8
  5. 2024Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimizationcited by 7
  6. 2026Towards Robust Scaling Laws for Optimizerscited by 4
  7. 2024Cost-Sensitive Multi-Fidelity Bayesian Optimization with Transfer of Learning Curve Extrapolationcited by 2
  8. 2025Tune My Adam, Please!cited by 2
  9. 2025Zero-Shot Performance Prediction for Probabilistic Scaling Lawscited by 2
  10. 2025Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuningcited by 2
  11. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
  12. 2026Auto Researching, not hyperparameter tuning: Convergence Analysis of 10,000 Experimentscited by 1
4 more
  1. 2025Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networkscited by 0
  2. 2025Efficient Hyperparameter Search for Non-Stationary Model Trainingcited by 0
  3. 2026Takeuchi's Information Criteria as Generalization Measures for DNNs Close to NTK Regimecited by 0
  4. 2026Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruningcited by 0