# PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning

arXiv: https://arxiv.org/abs/2306.12370 · HTML: https://mlsanity.com/paper/2306.12370/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 68 references that are in the map.
- First posted on arXiv 21 Jun 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://mlsanity.com/paper/2003.05997/) (2020)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [Zero-Cost Proxies for Lightweight NAS](https://mlsanity.com/paper/2101.08134/) (2021)
- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](https://mlsanity.com/paper/2007.04074/) (2020)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [Multi-fidelity Bayesian Optimisation with Continuous Approximations](https://mlsanity.com/paper/1703.06240/) (2017)
- [Batch Bayesian Optimization via Local Penalization](https://mlsanity.com/paper/1505.08052/) (2015)
- [On the State of the Art of Evaluation in Neural Language Models](https://mlsanity.com/paper/1707.05589/) (2017)
- [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014)
- [How Powerful are Performance Predictors in Neural Architecture Search?](https://mlsanity.com/paper/2104.01177/) (2021)
- [Few-Shot Bayesian Optimization with Deep Kernel Surrogates](https://mlsanity.com/paper/2101.07667/) (2021)
- [πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization](https://mlsanity.com/paper/2204.11051/) (2022)
- [Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL](https://mlsanity.com/paper/2006.13799/) (2020)
- [BOCK : Bayesian Optimization with Cylindrical Kernels](https://mlsanity.com/paper/1806.01619/) (2018)
- [Pre-trained Gaussian Processes for Bayesian Optimization](https://mlsanity.com/paper/2109.08215/) (2021)
- [On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning](https://mlsanity.com/paper/2102.13651/) (2021)
- [Bayesian Optimization in AlphaGo](https://mlsanity.com/paper/1812.06855/) (2018)
- [Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning](https://mlsanity.com/paper/1909.12552/) (2019)
- [Practical Transfer Learning for Bayesian Optimization](https://mlsanity.com/paper/1802.02219/) (2018)
- [Knowing The What But Not The Where in Bayesian Optimization](https://mlsanity.com/paper/1905.02685/) (2019)

## Built on by (most cited first, 30 of 31)

- [Large Language Model Agent for Hyper-Parameter Optimization](https://mlsanity.com/paper/2402.01881/) (2024)
- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization](https://mlsanity.com/paper/2404.16795/) (2024)
- [A General Framework for User-Guided Bayesian Optimization](https://mlsanity.com/paper/2311.14645/) (2023)
- [Position: A Call to Action for a Human-Centered AutoML Paradigm](https://mlsanity.com/paper/2406.03348/) (2024)
- [Benchmarking Deep Learning Models on NVIDIA Jetson Nano for Real-Time Systems: An Empirical Investigation](https://mlsanity.com/paper/2406.17749/) (2024)
- [Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning](https://mlsanity.com/paper/2309.03581/) (2023)
- [Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization](https://mlsanity.com/paper/2405.15393/) (2024)
- [Don't Waste Your Time: Early Stopping Cross-Validation](https://mlsanity.com/paper/2405.03389/) (2024)
- [Dual-view X-ray Detection: Can AI Detect Prohibited Items from Dual-view X-ray Images like Humans?](https://mlsanity.com/paper/2411.18082/) (2024)
- [Hyperparameter Optimization via Interacting with Probabilistic Circuits](https://mlsanity.com/paper/2505.17804/) (2025)
- [Optimizing Time Series Forecasting Architectures: A Hierarchical Neural Architecture Search Approach](https://mlsanity.com/paper/2406.05088/) (2024)
- [Auto-nnU-Net: Towards Automated Medical Image Segmentation](https://mlsanity.com/paper/2505.16561/) (2025)
- [carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks](https://mlsanity.com/paper/2506.06143/) (2025)
- [LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection](https://mlsanity.com/paper/2510.26510/) (2025)
- [LMEMs for post-hoc analysis of HPO Benchmarking](https://mlsanity.com/paper/2408.02533/) (2024)
- [Towards certifiable AI in aviation: landscape, challenges, and opportunities](https://mlsanity.com/paper/2409.08666/) (2024)
- [Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization](https://mlsanity.com/paper/2504.10735/) (2025)
- [CoTune: Co-evolutionary Configuration Tuning](https://mlsanity.com/paper/2509.24694/) (2025)
- [Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning](https://mlsanity.com/paper/2510.21379/) (2025)
- [Dynamic Priors in Bayesian Optimization for Hyperparameter Optimization](https://mlsanity.com/paper/2511.02570/) (2025)
- [DeepCAVE: A Visualization and Analysis Tool for Automated Machine Learning](https://mlsanity.com/paper/2512.01810/) (2025)
- [Multi-objective Hyperparameter Optimization in the Age of Deep Learning](https://mlsanity.com/paper/2511.08371/) (2025)
- [Strong convexity-guided hyper-parameter optimization for flatter losses](https://mlsanity.com/paper/2402.05025/) (2024)
- [Primitive Agentic First-Order Optimization](https://mlsanity.com/paper/2406.04841/) (2024)
- [Bootstrap Sampling Rate Greater than 1.0 May Improve Random Forest Performance](https://mlsanity.com/paper/2410.04297/) (2024)
- [XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML](https://mlsanity.com/paper/2508.00924/) (2025)
- [Quickly Tuning Foundation Models for Image Segmentation](https://mlsanity.com/paper/2508.17283/) (2025)
- [Iterated Population Based Training with Task-Agnostic Restarts](https://mlsanity.com/paper/2511.09190/) (2025)
- [Regime-Conditioned Evaluation in Multi-Context Bayesian Optimization](https://mlsanity.com/paper/2605.04895/) (2026)

## Abstract (arXiv)

Hyperparameters of Deep Learning (DL) pipelines are crucial for their downstream performance. While a large number of methods for Hyperparameter Optimization (HPO) have been developed, their incurred costs are often untenable for modern DL. Consequently, manual experimentation is still the most prevalent approach to optimize hyperparameters, relying on the researcher's intuition, domain knowledge, and cheap preliminary explorations. To resolve this misalignment between HPO algorithms and DL researchers, we propose PriorBand, an HPO algorithm tailored to DL, able to utilize both expert beliefs and cheap proxy tasks. Empirically, we demonstrate PriorBand's efficiency across a range of DL benchmarks and show its gains under informative expert input and robustness against poor expert beliefs