# SELECTOR: Selecting a Representative Benchmark Suite for Reproducible Statistical Comparison

arXiv: https://arxiv.org/abs/2204.11527 · HTML: https://mlsanity.com/paper/2204.11527/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2022.

## Built on by (most cited first, 11 of 11)

- [DynamoRep: Trajectory-Based Population Dynamics for Classification of Black-box Optimization Problems](https://mlsanity.com/paper/2306.05438/) (2023)
- [PS-AAS: Portfolio Selection for Automated Algorithm Selection in Black-Box Optimization](https://mlsanity.com/paper/2310.10685/) (2023)
- [Assessing the Generalizability of a Performance Predictive Model](https://mlsanity.com/paper/2306.00040/) (2023)
- [Generalization Ability of Feature-based Performance Prediction Models: A Statistical Analysis across Benchmarks](https://mlsanity.com/paper/2405.12259/) (2024)
- [RF+clust for Leave-One-Problem-Out Performance Prediction](https://mlsanity.com/paper/2301.09524/) (2023)
- [TransOpt: Transformer-based Representation Learning for Optimization Problem Classification](https://mlsanity.com/paper/2311.18035/) (2023)
- [A Survey of Features Used for Representing Black-box Single-objective Continuous Optimization](https://mlsanity.com/paper/2406.06629/) (2024)
- [carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks](https://mlsanity.com/paper/2506.06143/) (2025)
- [Consistent and Distinctive: LLM Benchmark Efficiency via Maximum Independent Set Prompt Selection on Similarity Graphs](https://mlsanity.com/paper/2606.01400/) (2026)
- [Frugal Algorithm Selection](https://mlsanity.com/paper/2405.11059/) (2024)
- [Unsupervised Multi-kernel Learning for Automated Algorithm Selection](https://mlsanity.com/paper/2607.19031/) (2026)

## Abstract (arXiv)

Fair algorithm evaluation is conditioned on the existence of high-quality benchmark datasets that are non-redundant and are representative of typical optimization scenarios. In this paper, we evaluate three heuristics for selecting diverse problem instances which should be involved in the comparison of optimization algorithms in order to ensure robust statistical algorithm performance analysis. The first approach employs clustering to identify similar groups of problem instances and subsequent sampling from each cluster to construct new benchmarks, while the other two approaches use graph algorithms for identifying dominating and maximal independent sets of nodes. We demonstrate the applicability of the proposed heuristics by performing a statistical performance analysis of five portfolios consisting of three optimization algorithms on five of the most commonly used optimization benchmarks. The results indicate that the statistical analyses of the algorithms' performance, conducted on each benchmark separately, produce conflicting outcomes, which can be used to give a false indication of the superiority of one algorithm over another. On the other hand, when the analysis is conducted on the problem instances selected with the proposed heuristics, which uniformly cover the problem landscape, the statistical outcomes are robust and consistent.