# FastBO: Fast HPO and NAS with Adaptive Fidelity Identification

arXiv: https://arxiv.org/abs/2409.00584 · HTML: https://mlsanity.com/paper/2409.00584/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 1 Sep 2024.

## Builds on (most cited first, 29 of 29)

- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014)
- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization](https://mlsanity.com/paper/2204.11051/) (2022)
- [BOCK : Bayesian Optimization with Cylindrical Kernels](https://mlsanity.com/paper/1806.01619/) (2018)
- [Bayesian Optimization with Gradients](https://mlsanity.com/paper/1703.04389/) (2017)
- [Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization](https://mlsanity.com/paper/1905.04970/) (2019)
- [Explaining Hyperparameter Optimization via Partial Dependence Plots](https://mlsanity.com/paper/2111.04820/) (2021)
- [Scaling Gaussian Processes with Derivative Information Using Variational Inference](https://mlsanity.com/paper/2107.04061/) (2021)
- [Supervising the Multi-Fidelity Race of Hyperparameter Configurations](https://mlsanity.com/paper/2202.09774/) (2022)
- [Accelerating Experimental Design by Incorporating Experimenter Hunches](https://mlsanity.com/paper/1907.09065/) (2019)
- [Unbounded Bayesian Optimization via Regularization](https://mlsanity.com/paper/1508.03666/) (2015)
- [Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale](https://mlsanity.com/paper/2201.06834/) (2022)
- [PASHA: Efficient HPO and NAS with Progressive Resource Allocation](https://mlsanity.com/paper/2207.06940/) (2022)
- [Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning](https://mlsanity.com/paper/1801.01596/) (2018)
- [Scalable First-Order Bayesian Optimization via Structured Automatic Differentiation](https://mlsanity.com/paper/2206.08366/) (2022)
- [Optimizing Hyperparameters with Conformal Quantile Regression](https://mlsanity.com/paper/2305.03623/) (2023)
- [Regulating Model Reliance on Non-Robust Features by Smoothing Input Marginal Density](https://mlsanity.com/paper/2407.04370/) (2024)
- [Fast Parallel Exact Inference on Bayesian Networks: Poster](https://mlsanity.com/paper/2212.04241/) (2022)
- [Fast Parallel Bayesian Network Structure Learning](https://mlsanity.com/paper/2212.04259/) (2022)
- [Improving Accuracy of Interpretability Measures in Hyperparameter Optimization via Bayesian Algorithm Execution](https://mlsanity.com/paper/2206.05447/) (2022)
- [A Backdoor-based Explainable AI Benchmark for High Fidelity Evaluation of Attributions](https://mlsanity.com/paper/2405.02344/) (2024)
- [Fast-PGM: Fast Probabilistic Graphical Model Learning and Inference](https://mlsanity.com/paper/2405.15605/) (2024)

## Built on by (most cited first, 1 of 1)

- [Budget Allocation for Unknown Value Functions in a Lipschitz Space](https://mlsanity.com/paper/2510.10605/) (2025)

## Abstract (arXiv)

Hyperparameter optimization (HPO) and neural architecture search (NAS) are powerful in attaining state-of-the-art machine learning models, with Bayesian optimization (BO) standing out as a mainstream method. Extending BO into the multi-fidelity setting has been an emerging research topic, but faces the challenge of determining an appropriate fidelity for each hyperparameter configuration to fit the surrogate model. To tackle the challenge, we propose a multi-fidelity BO method named FastBO, which adaptively decides the fidelity for each configuration and efficiently offers strong performance. The advantages are achieved based on the novel concepts of efficient point and saturation point for each configuration.We also show that our adaptive fidelity identification strategy provides a way to extend any single-fidelity method to the multi-fidelity setting, highlighting its generality and applicability.