# Improving Accuracy of Interpretability Measures in Hyperparameter Optimization via Bayesian Algorithm Execution

arXiv: https://arxiv.org/abs/2206.05447 · HTML: https://mlsanity.com/paper/2206.05447/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 46 references that are in the map.
- First posted on arXiv 11 Jun 2022.

## Builds on (most cited first, 18 of 18)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Predictive Entropy Search for Efficient Global Optimization of Black-box Functions](https://mlsanity.com/paper/1406.2541/) (2014)
- [Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020](https://mlsanity.com/paper/2104.10201/) (2021)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [GPflow: A Gaussian process library using TensorFlow](https://mlsanity.com/paper/1610.08733/) (2016)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization](https://mlsanity.com/paper/2204.11051/) (2022)
- [Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL](https://mlsanity.com/paper/2006.13799/) (2020)
- [Hyperparameter Importance Across Datasets](https://mlsanity.com/paper/1710.04725/) (2017)
- [Bayesian Optimization with Gradients](https://mlsanity.com/paper/1703.04389/) (2017)
- [Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search](https://mlsanity.com/paper/1807.06906/) (2018)
- [Explaining Hyperparameter Optimization via Partial Dependence Plots](https://mlsanity.com/paper/2111.04820/) (2021)
- [Bayesian Algorithm Execution: Estimating Computable Properties of Black-box Functions Using Mutual Information](https://mlsanity.com/paper/2104.09460/) (2021)
- [Learning Interpretable Models Through Multi-Objective Neural Architecture Search](https://mlsanity.com/paper/2112.08645/) (2021)
- [Automatic Componentwise Boosting: An Interpretable AutoML System](https://mlsanity.com/paper/2109.05583/) (2021)

## Built on by (most cited first, 2 of 2)

- [FastBO: Fast HPO and NAS with Adaptive Fidelity Identification](https://mlsanity.com/paper/2409.00584/) (2024)
- [Explaining AutoClustering: Uncovering Meta-Feature Contribution in AutoML for Clustering](https://mlsanity.com/paper/2602.18348/) (2026)

## Abstract (arXiv)

Despite all the benefits of automated hyperparameter optimization (HPO), most modern HPO algorithms are black-boxes themselves. This makes it difficult to understand the decision process which leads to the selected configuration, reduces trust in HPO, and thus hinders its broad adoption. Here, we study the combination of HPO with interpretable machine learning (IML) methods such as partial dependence plots. These techniques are more and more used to explain the marginal effect of hyperparameters on the black-box cost function or to quantify the importance of hyperparameters. However, if such methods are naively applied to the experimental data of the HPO process in a post-hoc manner, the underlying sampling bias of the optimizer can distort interpretations. We propose a modified HPO method which efficiently balances the search for the global optimum w.r.t. predictive performance and the reliable estimation of IML explanations of an underlying black-box function by coupling Bayesian optimization and Bayesian Algorithm Execution. On benchmark cases of both synthetic objectives and HPO of a neural network, we demonstrate that our method returns more reliable explanations of the underlying black-box without a loss of optimization performance.