# HPO X ELA: Investigating Hyperparameter Optimization Landscapes by Means of Exploratory Landscape Analysis

arXiv: https://arxiv.org/abs/2208.00220 · HTML: https://mlsanity.com/paper/2208.00220/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 40 references that are in the map.
- First posted on arXiv 30 Jul 2022.

## Builds on (most cited first, 10 of 10)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [Automated Algorithm Selection on Continuous Black-Box Problems By Combining Exploratory Landscape Analysis and Machine Learning](https://mlsanity.com/paper/1711.08921/) (2017)
- [Comprehensive Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems Using the R-Package flacco](https://mlsanity.com/paper/1708.05258/) (2017)
- [Revisiting Bayesian Optimization in the light of the COCO benchmark](https://mlsanity.com/paper/2103.16649/) (2021)
- [Fitness Landscape Footprint: A Framework to Compare Neural Architecture Search Problems](https://mlsanity.com/paper/2111.01584/) (2021)

## Built on by (most cited first, 10 of 10)

- [AutoRL Hyperparameter Landscapes](https://mlsanity.com/paper/2304.02396/) (2023)
- [On the Hyperparameter Loss Landscapes of Machine Learning Models: An Exploratory Study](https://mlsanity.com/paper/2311.14014/) (2023)
- [HyperSHAP: Shapley Values and Interactions for Explaining Hyperparameter Optimization](https://mlsanity.com/paper/2502.01276/) (2025)
- [A Survey of Features Used for Representing Black-box Single-objective Continuous Optimization](https://mlsanity.com/paper/2406.06629/) (2024)
- [Tools for Landscape Analysis of Optimisation Problems in Procedural Content Generation for Games](https://mlsanity.com/paper/2302.08479/) (2023)
- [Towards the Inferrence of Structural Similarity of Combinatorial Landscapes](https://mlsanity.com/paper/2312.02720/) (2023)
- [Greedy Restart Schedules: A Baseline for Dynamic Algorithm Selection on Numerical Black-box Optimization Problems](https://mlsanity.com/paper/2504.11440/) (2025)
- [Benchmarking MOEAs for solving continuous multi-objective RL problems](https://mlsanity.com/paper/2505.13726/) (2025)
- [Characterizing Fitness Landscape Structures in Prompt Engineering](https://mlsanity.com/paper/2509.05375/) (2025)
- [Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Network](https://mlsanity.com/paper/2512.16491/) (2025)

## Abstract (arXiv)

Hyperparameter optimization (HPO) is a key component of machine learning models for achieving peak predictive performance. While numerous methods and algorithms for HPO have been proposed over the last years, little progress has been made in illuminating and examining the actual structure of these black-box optimization problems. Exploratory landscape analysis (ELA) subsumes a set of techniques that can be used to gain knowledge about properties of unknown optimization problems. In this paper, we evaluate the performance of five different black-box optimizers on 30 HPO problems, which consist of two-, three- and five-dimensional continuous search spaces of the XGBoost learner trained on 10 different data sets. This is contrasted with the performance of the same optimizers evaluated on 360 problem instances from the black-box optimization benchmark (BBOB). We then compute ELA features on the HPO and BBOB problems and examine similarities and differences. A cluster analysis of the HPO and BBOB problems in ELA feature space allows us to identify how the HPO problems compare to the BBOB problems on a structural meta-level. We identify a subset of BBOB problems that are close to the HPO problems in ELA feature space and show that optimizer performance is comparably similar on these two sets of benchmark problems. We highlight open challenges of ELA for HPO and discuss potential directions of future research and applications.