# Black-Box Optimization Revisited: Improving Algorithm Selection Wizards through Massive Benchmarking

arXiv: https://arxiv.org/abs/2010.04542 · HTML: https://mlsanity.com/paper/2010.04542/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2020.

## Built on by (most cited first, 14 of 14)

- [TTOpt: A Maximum Volume Quantized Tensor Train-based Optimization and its Application to Reinforcement Learning](https://mlsanity.com/paper/2205.00293/) (2022)
- [To Switch or not to Switch: Predicting the Benefit of Switching between Algorithms based on Trajectory Features](https://mlsanity.com/paper/2302.09075/) (2023)
- [PS-AAS: Portfolio Selection for Automated Algorithm Selection in Black-Box Optimization](https://mlsanity.com/paper/2310.10685/) (2023)
- [Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym](https://mlsanity.com/paper/2310.00077/) (2023)
- [Learning Discrete Structured Variational Auto-Encoder using Natural Evolution Strategies](https://mlsanity.com/paper/2205.01324/) (2022)
- [Evolutionary Retrofitting](https://mlsanity.com/paper/2410.11330/) (2024)
- [Exploiting Function-Family Structure in Analog Circuit Optimization](https://mlsanity.com/paper/2512.00712/) (2025)
- [Learning-To-Ensemble by Contextual Rank Aggregation in E-Commerce](https://mlsanity.com/paper/2107.08598/) (2021)
- [Log-normal Mutations and their Use in Detecting Surreptitious Fake Images](https://mlsanity.com/paper/2409.15119/) (2024)
- [Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Training](https://mlsanity.com/paper/2507.01752/) (2025)
- [Validated respiratory drug deposition predictions from 2D and 3D medical images with statistical shape models and convolutional neural networks](https://mlsanity.com/paper/2303.01036/) (2023)
- [Primitive Agentic First-Order Optimization](https://mlsanity.com/paper/2406.04841/) (2024)
- [Towards a Distributed Federated Learning Aggregation Placement using Particle Swarm Intelligence](https://mlsanity.com/paper/2504.16227/) (2025)
- [Reinforcement learning to choose optimizers](https://mlsanity.com/paper/2609.01811/) (2026)

## Abstract (arXiv)

Existing studies in black-box optimization for machine learning suffer from low generalizability, caused by a typically selective choice of problem instances used for training and testing different optimization algorithms. Among other issues, this practice promotes overfitting and poor-performing user guidelines. To address this shortcoming, we propose in this work a benchmark suite, OptimSuite, which covers a broad range of black-box optimization problems, ranging from academic benchmarks to real-world applications, from discrete over numerical to mixed-integer problems, from small to very large-scale problems, from noisy over dynamic to static problems, etc. We demonstrate the advantages of such a broad collection by deriving from it Automated Black Box Optimizer (ABBO), a general-purpose algorithm selection wizard. Using three different types of algorithm selection techniques, ABBO achieves competitive performance on all benchmark suites. It significantly outperforms previous state of the art on some of them, including YABBOB and LSGO. ABBO relies on many high-quality base components. Its excellent performance is obtained without any task-specific parametrization. The OptimSuite benchmark collection, the ABBO wizard and its base solvers have all been merged into the open-source Nevergrad platform, where they are available for reproducible research.