mlsanityData to 14 Oct 2026

HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO

arXiv 2109.06716 ↗First posted 14 Sep 2021cs.LGcited by 112

Not on the map. Why?

Abstract (arXiv)

To achieve peak predictive performance, hyperparameter optimization (HPO) is a crucial component of machine learning and its applications. Over the last years, the number of efficient algorithms and tools for HPO grew substantially. At the same time, the community is still lacking realistic, diverse, computationally cheap, and standardized benchmarks. This is especially the case for multi-fidelity HPO methods. To close this gap, we propose HPOBench, which includes 7 existing and 5 new benchmark families, with a total of more than 100 multi-fidelity benchmark problems. HPOBench allows to run this extendable set of multi-fidelity HPO benchmarks in a reproducible way by isolating and packaging the individual benchmarks in containers. It also provides surrogate and tabular benchmarks for computationally affordable yet statistically sound evaluations. To demonstrate HPOBench's broad compatibility with various optimization tools, as well as its usefulness, we conduct an exemplary large-scale study evaluating 13 optimizers from 6 optimization tools. We provide HPOBench here: https://github.com/automl/HPOBench.

Key facts. Cited by 112 papers in the map (data through 14 Oct 2026). Builds on 54 of its 121 references that are in the map. First posted on arXiv 14 Sep 2021.

Builds on

54 of 121 references.

References · 54

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  3. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  4. 2016OpenAI Gymcited by 1,736
  5. 2018Datasheets for Datasetscited by 1,386
  6. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  7. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  8. 2014OpenML: networked science in machine learningcited by 526
  9. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  10. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
  11. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 374
  12. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
38 more
  1. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 344
  2. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 343
  3. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 221
  4. 2018A System for Massively Parallel Hyperparameter Tuningcited by 202
  5. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  6. 2019Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approachescited by 185
  7. 2017OpenML Benchmarking Suitescited by 178
  8. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 148
  9. 2016COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Settingcited by 138
  10. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 132
  11. 2017Multi-fidelity Bayesian Optimisation with Continuous Approximationscited by 121
  12. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 104
  13. 2019Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuningcited by 93
  14. 2014Freeze-Thaw Bayesian Optimizationcited by 90
  15. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  16. 2019An Open Source AutoML Benchmarkcited by 85
  17. 2020HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisationcited by 83
  18. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  19. 2019Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonflycited by 67
  20. 2016Theano: A Python framework for fast computation of mathematical expressionscited by 64
  21. 2020NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Searchcited by 64
  22. 2017Bayesian Optimization with Gradientscited by 59
  23. 2019Tabular Benchmarks for Joint Architecture and Hyperparameter Optimizationcited by 58
  24. 2019Best Practices for Scientific Research on Neural Architecture Searchcited by 54
  25. 2018A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processescited by 52
  26. 2020Benchmarking in Optimization: Best Practice and Open Issuescited by 38
  27. 2019Mixed-Variable Bayesian Optimizationcited by 29
  28. 2020Using a thousand optimization tasks to learn hyperparameter search strategiescited by 29
  29. 2020Olympus: a benchmarking framework for noisy optimization and experiment planningcited by 29
  30. 2018IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristicscited by 28
  31. 2017Pitfalls and Best Practices in Algorithm Configurationcited by 25
  32. 2019Meta-Surrogate Benchmarking for Hyperparameter Optimizationcited by 17
  33. 2019Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameterscited by 16
  34. 2020Sherpa: Robust Hyperparameter Optimization for Machine Learningcited by 14
  35. 2018Automatic Exploration of Machine Learning Experiments on OpenMLcited by 13
  36. 2018Terminologies for Reproducible Researchcited by 11
  37. 2020Differential Evolution for Neural Architecture Searchcited by 10
  38. 2016Funneled Bayesian Optimization for Design, Tuning and Control of Autonomous Systemscited by 9

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 15; 2023: 25; 2024: 23; 2025: 17; 2026: 28.

112 papers.

Sort:

By area:

  1. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 230
  2. 2023Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performancecited by 147
  3. 2024Large Language Models to Enhance Bayesian Optimizationcited by 135
  4. 2023PFNs4BO: In-Context Learning for Bayesian Optimizationcited by 73
  5. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  6. 2022πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimizationcited by 70
  7. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 69
  8. 2024Large Language Model Agent for Hyper-Parameter Optimizationcited by 64
  9. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  10. 2022Joint Entropy Search for Maximally-Informed Bayesian Optimizationcited by 55
  11. 2022AMLB: an AutoML Benchmarkcited by 55
  12. 2021LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lassocited by 41
38 more
  1. 2022Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overviewcited by 37
  2. 2022NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easycited by 36
  3. 2023TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applicationscited by 33
  4. 2023End-to-End Meta-Bayesian Optimisation with Transformer Neural Processescited by 32
  5. 2024Kolb-Based Experiential Learning for Generalist Agents with Human-Level Kaggle Data Science Performancecited by 25
  6. 2024Toward Automated Algorithm Design: A Survey and Practical Guide to Meta-Black-Box-Optimizationcited by 24
  7. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  8. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  9. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
  10. 2025Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMscited by 17
  11. 2023Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimizationcited by 16
  12. 2025Offline Model-Based Optimization: Comprehensive Reviewcited by 15
  13. 2025OptunaHub: A Platform for Black-Box Optimizationcited by 14
  14. 2021Automated Deep Learning: Neural Architecture Search Is Not the Endcited by 13
  15. 2022Batch Bayesian optimisation via density-ratio estimation with guaranteescited by 9
  16. 2024Hyperparameter Importance Analysis for Multi-Objective AutoMLcited by 9
  17. 2024What Do Machine Learning Researchers Mean by "Reproducible"?cited by 9
  18. 2022Task Selection for AutoML System Evaluationcited by 7
  19. 2022Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimatorcited by 7
  20. 2024Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimizationcited by 7
  21. 2022FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimizationcited by 6
  22. 2022Asynchronous Decentralized Bayesian Optimization for Large Scale Hyperparameter Optimizationcited by 6
  23. 2023Multi-Fidelity Multi-Armed Bandits Revisitedcited by 6
  24. 2023Is One Epoch All You Need For Multi-Fidelity Hyperparameter Optimization?cited by 6
  25. 2024Multi-Fidelity Methods for Optimization: A Surveycited by 6
  26. 2025The Gittins Index: A Design Principle for Decision-Making Under Uncertaintycited by 6
  27. 2026LABO: LLM-Accelerated Bayesian Optimization through Broad Exploration and Selective Experimentationcited by 6
  28. 2022General Cyclical Training of Neural Networkscited by 5
  29. 2023A Coreset Learning Reality Checkcited by 5
  30. 2023Self-Adjusting Weighted Expected Improvement for Bayesian Optimizationcited by 5
  31. 2024The Unreasonable Effectiveness Of Early Discarding After One Epoch In Neural Network Hyperparameter Optimizationcited by 5
  32. 2022HyperBO+: Pre-training a universal prior for Bayesian optimization with hierarchical Gaussian processescited by 4
  33. 2023On the Hyperparameter Loss Landscapes of Machine Learning Models: An Exploratory Studycited by 4
  34. 2022B2EA: An Evolutionary Algorithm Assisted by Two Bayesian Optimization Modules for Neural Architecture Searchcited by 3
  35. 2023Scalable Meta-Learning with Gaussian Processescited by 3
  36. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learningcited by 3
  37. 2024A Human-in-the-Loop Fairness-Aware Model Selection Framework for Complex Fairness Objective Landscapescited by 3
  38. 2025carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarkscited by 3

50 most cited of 112 All 112 (JSON)