# Benchmarking Feature-based Algorithm Selection Systems for Black-box Numerical Optimization

arXiv: https://arxiv.org/abs/2109.08377 · HTML: https://mlsanity.com/paper/2109.08377/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2021.

## Built on by (most cited first, 7 of 7)

- [Landscape Features in Single-Objective Continuous Optimization: Have We Hit a Wall in Algorithm Selection Generalization?](https://mlsanity.com/paper/2501.17663/) (2025)
- [A Survey of Features Used for Representing Black-box Single-objective Continuous Optimization](https://mlsanity.com/paper/2406.06629/) (2024)
- [The Pitfalls of Benchmarking in Algorithm Selection: What We Are Getting Wrong](https://mlsanity.com/paper/2505.07750/) (2025)
- [Improving Algorithm-Selection and Performance-Prediction via Learning Discriminating Training Samples](https://mlsanity.com/paper/2404.05359/) (2024)
- [Towards the Inferrence of Structural Similarity of Combinatorial Landscapes](https://mlsanity.com/paper/2312.02720/) (2023)
- [Identifying Easy Instances to Improve Efficiency of ML Pipelines for Algorithm-Selection](https://mlsanity.com/paper/2406.16999/) (2024)
- [On Constructing Algorithm Portfolios in Algorithm Selection for Computationally Expensive Black-box Optimization in the Fixed-budget Setting](https://mlsanity.com/paper/2405.10976/) (2024)

## Abstract (arXiv)

Feature-based algorithm selection aims to automatically find the best one from a portfolio of optimization algorithms on an unseen problem based on its landscape features. Feature-based algorithm selection has recently received attention in the research field of black-box numerical optimization. However, there is still room for analysis of algorithm selection for black-box optimization. Most previous studies have focused only on whether an algorithm selection system can outperform the single-best solver in a portfolio. In addition, a benchmarking methodology for algorithm selection systems has not been well investigated in the literature. In this context, this paper analyzes algorithm selection systems on the 24 noiseless black-box optimization benchmarking functions. First, we demonstrate that the first successful performance measure is more reliable than the expected runtime measure for benchmarking algorithm selection systems. Then, we examine the influence of randomness on the performance of algorithm selection systems. We also show that the performance of algorithm selection systems can be significantly improved by using sequential least squares programming as a pre-solver. We point out that the difficulty of outperforming the single-best solver depends on algorithm portfolios, cross-validation methods, and dimensions. Finally, we demonstrate that the effectiveness of algorithm portfolios depends on various factors. These findings provide fundamental insights for algorithm selection for black-box optimization.