# GPU Accelerated Exhaustive Search for Optimal Ensemble of Black-Box Optimization Algorithms

arXiv: https://arxiv.org/abs/2012.04201 · HTML: https://mlsanity.com/paper/2012.04201/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2020.

## Built on by (most cited first, 5 of 5)

- [Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020](https://mlsanity.com/paper/2104.10201/) (2021)
- [A survey on multi-objective hyperparameter optimization algorithms for Machine Learning](https://mlsanity.com/paper/2111.13755/) (2021)
- [Learning How to Optimize Black-Box Functions With Extreme Limits on the Number of Function Evaluations](https://mlsanity.com/paper/2103.10321/) (2021)
- [Zero-shot Task Transfer for Invoice Extraction via Class-aware QA Ensemble](https://mlsanity.com/paper/2108.06069/) (2021)
- [Trading Off Resource Budgets for Improved Regret Bounds](https://mlsanity.com/paper/2210.05789/) (2022)

## Abstract (arXiv)

Black-box optimization is essential for tuning complex machine learning algorithms which are easier to experiment with than to understand. In this paper, we show that a simple ensemble of black-box optimization algorithms can outperform any single one of them. However, searching for such an optimal ensemble requires a large number of experiments. We propose a Multi-GPU-optimized framework to accelerate a brute force search for the optimal ensemble of black-box optimization algorithms by running many experiments in parallel. The lightweight optimizations are performed by CPU while expensive model training and evaluations are assigned to GPUs. We evaluate 15 optimizers by training 2.7 million models and running 541,440 optimizations. On a DGX-1, the search time is reduced from more than 10 days on two 20-core CPUs to less than 24 hours on 8-GPUs. With the optimal ensemble found by GPU-accelerated exhaustive search, we won the 2nd place of NeurIPS 2020 black-box optimization challenge.