# Learning How to Optimize Black-Box Functions With Extreme Limits on the Number of Function Evaluations

arXiv: https://arxiv.org/abs/2103.10321 · HTML: https://mlsanity.com/paper/2103.10321/

## Key facts

- Cited by 5 papers in the map (data through 7 Oct 2026).
- Builds on 9 of its 42 references that are in the map.
- First posted on arXiv 18 Mar 2021.

## Builds on (most cited first, 9 of 9)

- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://mlsanity.com/paper/1603.08785/) (2016)
- [Few-Shot Bayesian Optimization with Deep Kernel Surrogates](https://mlsanity.com/paper/2101.07667/) (2021)
- [ASlib: A Benchmark Library for Algorithm Selection](https://mlsanity.com/paper/1506.02465/) (2015)
- [TopRank: A practical algorithm for online stochastic ranking](https://mlsanity.com/paper/1806.02248/) (2018)
- [GPU Accelerated Exhaustive Search for Optimal Ensemble of Black-Box Optimization Algorithms](https://mlsanity.com/paper/2012.04201/) (2020)
- [Online Preselection with Context Information under the Plackett-Luce Model](https://mlsanity.com/paper/2002.04275/) (2020)
- [Bayesian task embedding for few-shot Bayesian optimization](https://mlsanity.com/paper/2001.00637/) (2020)

## Built on by (most cited first, 5 of 5)

- [E2ETune: End-to-End Knob Tuning via Fine-tuned Generative Language Model](https://mlsanity.com/paper/2404.11581/) (2024)
- [Optimizing Algorithms From Pairwise User Preferences](https://mlsanity.com/paper/2308.04571/) (2023)
- [The importance of being constrained: dealing with infeasible solutions in Differential Evolution and beyond](https://mlsanity.com/paper/2203.03512/) (2022)
- [Discovering Multiple Algorithm Configurations](https://mlsanity.com/paper/2303.07434/) (2023)
- [When AI Bends Metal: AI-Assisted Optimization of Design Parameters in Sheet Metal Forming](https://mlsanity.com/paper/2511.22302/) (2025)

## Abstract (arXiv)

We consider black-box optimization in which only an extremely limited number of function evaluations, on the order of around 100, are affordable and the function evaluations must be performed in even fewer batches of a limited number of parallel trials. This is a typical scenario when optimizing variable settings that are very costly to evaluate, for example in the context of simulation-based optimization or machine learning hyperparameterization. We propose an original method that uses established approaches to propose a set of points for each batch and then down-selects from these candidate points to the number of trials that can be run in parallel. The key novelty of our approach lies in the introduction of a hyperparameterized method for down-selecting the number of candidates to the allowed batch-size, which is optimized offline using automated algorithm configuration. We tune this method for black box optimization and then evaluate on classical black box optimization benchmarks. Our results show that it is possible to learn how to combine evaluation points suggested by highly diverse black box optimization methods conditioned on the progress of the optimization. Compared with the state of the art in black box minimization and various other methods specifically geared towards few-shot minimization, we achieve an average reduction of 50% of normalized cost, which is a highly significant improvement in performance.