# Meta-Surrogate Benchmarking for Hyperparameter Optimization

arXiv: https://arxiv.org/abs/1905.12982 · HTML: https://mlsanity.com/paper/1905.12982/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2019.

## Built on by (most cited first, 17 of 17)

- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization](https://mlsanity.com/paper/2204.11051/) (2022)
- [YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization](https://mlsanity.com/paper/2109.03670/) (2021)
- [AMLB: an AutoML Benchmark](https://mlsanity.com/paper/2207.12560/) (2022)
- [LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso](https://mlsanity.com/paper/2111.02790/) (2021)
- [BORE: Bayesian Optimization by Density-Ratio Estimation](https://mlsanity.com/paper/2102.09009/) (2021)
- [Meta Learning Black-Box Population-Based Optimizers](https://mlsanity.com/paper/2103.03526/) (2021)
- [Learning to Transfer for Evolutionary Multitasking](https://mlsanity.com/paper/2406.14359/) (2024)
- [ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning](https://mlsanity.com/paper/2409.18827/) (2024)
- [Rethinking Performance Analysis for Configurable Software Systems: A Case Study from a Fitness Landscape Perspective](https://mlsanity.com/paper/2412.16888/) (2024)
- [Model Selection for Production System via Automated Online Experiments](https://mlsanity.com/paper/2105.13420/) (2021)
- [CATBench: A Compiler Autotuning Benchmarking Suite for Black-box Optimization](https://mlsanity.com/paper/2406.17811/) (2024)
- [Natural Evolutionary Search meets Probabilistic Numerics](https://mlsanity.com/paper/2507.07288/) (2025)
- [Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining](https://mlsanity.com/paper/2601.22108/) (2026)
- [AutoMC: Automated Model Compression based on Domain Knowledge and Progressive search strategy](https://mlsanity.com/paper/2201.09884/) (2022)
- [An Open-Source Training Dataset for Foundation Models for Black-box Optimization](https://mlsanity.com/paper/2605.23417/) (2026)

## Abstract (arXiv)

Despite the recent progress in hyperparameter optimization (HPO), available benchmarks that resemble real-world scenarios consist of a few and very large problem instances that are expensive to solve. This blocks researchers and practitioners not only from systematically running large-scale comparisons that are needed to draw statistically significant results but also from reproducing experiments that were conducted before. This work proposes a method to alleviate these issues by means of a meta-surrogate model for HPO tasks trained on off-line generated data. The model combines a probabilistic encoder with a multi-task model such that it can generate inexpensive and realistic tasks of the class of problems of interest. We demonstrate that benchmarking HPO methods on samples of the generative model allows us to draw more coherent and statistically significant conclusions that can be reached orders of magnitude faster than using the original tasks. We provide evidence of our findings for various HPO methods on a wide class of problems.