mlsanityData to 7 Oct 2026

YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization

arXiv 2109.03670 ↗First posted 8 Sep 2021cs.LG stat.MLcited by 61

Not on the map. Why?

Abstract (arXiv)

When developing and analyzing new hyperparameter optimization methods, it is vital to empirically evaluate and compare them on well-curated benchmark suites. In this work, we propose a new set of challenging and relevant benchmark problems motivated by desirable properties and requirements for such benchmarks. Our new surrogate-based benchmark collection consists of 14 scenarios that in total constitute over 700 multi-fidelity hyperparameter optimization problems, which all enable multi-objective hyperparameter optimization. Furthermore, we empirically compare surrogate-based benchmarks to the more widely-used tabular benchmarks, and demonstrate that the latter may produce unfaithful results regarding the performance ranking of HPO methods. We examine and compare our benchmark collection with respect to defined requirements and propose a single-objective as well as a multi-objective benchmark suite on which we compare 7 single-objective and 7 multi-objective optimizers in a benchmark experiment. Our software is available at [https://github.com/slds-lmu/yahpo_gym].

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 32 of its 105 references that are in the map. First posted on arXiv 8 Sep 2021.

Builds on

32 of 105 references.

References · 32

  1. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  3. 2016Wide & Deep Learning for Recommender Systemscited by 745
  4. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  5. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  6. 2019Green AIcited by 620
  7. 2014OpenML: networked science in machine learningcited by 523
  8. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  9. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  10. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 219
  11. 2018A System for Massively Parallel Hyperparameter Tuningcited by 201
  12. 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 175
20 more
  1. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  2. 2016COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Settingcited by 135
  3. 2021The Benchmark Lotterycited by 133
  4. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 131
  5. 2017Multi-fidelity Bayesian Optimisation with Continuous Approximationscited by 120
  6. 2014Freeze-Thaw Bayesian Optimizationcited by 86
  7. 2016Entity Embeddings of Categorical Variablescited by 85
  8. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  9. 2019Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimizationcited by 70
  10. 2020Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDLcited by 67
  11. 2015ASlib: A Benchmark Library for Algorithm Selectioncited by 36
  12. 2018Practical Transfer Learning for Bayesian Optimizationcited by 26
  13. 2019BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameterscited by 25
  14. 2019Quantifying Model Complexity via Functional Decomposition for Better Post-Hoc Interpretabilitycited by 22
  15. 2021Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimizationcited by 19
  16. 2019Meta-Surrogate Benchmarking for Hyperparameter Optimizationcited by 16
  17. 2019Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameterscited by 16
  18. 2021Over-optimism in benchmark studies and the multiplicity of design and analysis options when interpreting their resultscited by 15
  19. 2018Learning Multiple Defaults for Machine Learning Algorithmscited by 10
  20. 2021Meta-Learning for Symbolic Hyperparameter Defaultscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 10; 2024: 10; 2025: 16; 2026: 20.

61 papers.

Sort:

By area:

  1. 2022Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overviewcited by 37
  2. 2023Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimizationcited by 15
  3. 2025OptunaHub: A Platform for Black-Box Optimizationcited by 14
  4. 2023Optimizing Hyperparameters with Conformal Quantile Regressioncited by 10
  5. 2025BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RLcited by 9
  6. 2025Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Frameworkcited by 8
  7. 2024Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimizationcited by 7
  8. 2022FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimizationcited by 6
  9. 2023Multi-Fidelity Multi-Armed Bandits Revisitedcited by 6
  10. 2023Is One Epoch All You Need For Multi-Fidelity Hyperparameter Optimization?cited by 6
  11. 2024Multi-Fidelity Methods for Optimization: A Surveycited by 5
  12. 2024The Unreasonable Effectiveness Of Early Discarding After One Epoch In Neural Network Hyperparameter Optimizationcited by 5
38 more
  1. 2025HyperSHAP: Shapley Values and Interactions for Explaining Hyperparameter Optimizationcited by 4
  2. 2025Put CASH on Bandits: A Max K-Armed Problem for Automated Machine Learningcited by 4
  3. 2022A Collection of Quality Diversity Optimization Problems Derived from Hyperparameter Optimization of Machine Learning Modelscited by 3
  4. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learningcited by 3
  5. 2025carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarkscited by 3
  6. 2026SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMscited by 3
  7. 2022Towards Meta-learned Algorithm Selection using Implicit Fidelity Informationcited by 2
  8. 2023MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimizationcited by 2
  9. 2023Python Wrapper for Simulating Multi-Fidelity Optimization on HPO Benchmarks without Any Waitcited by 2
  10. 2024Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarkscited by 2
  11. 2024LMEMs for post-hoc analysis of HPO Benchmarkingcited by 2
  12. 2025Dynamic Priors in Bayesian Optimization for Hyperparameter Optimizationcited by 2
  13. 2023Obeying the Order: Introducing Ordered Transfer Hyperparameter Optimisationcited by 1
  14. 2023Parallel Multi-Objective Hyperparameter Optimization with Uniform Normalization and Bounded Objectivescited by 1
  15. 2023Datasets and Benchmarks for Nanophotonic Structure and Parametric Design Simulationscited by 1
  16. 2024Is Hyper-Parameter Optimization Different for Software Analytics?cited by 1
  17. 2024A Trajectory-Based Bayesian Approach to Multi-Objective Hyperparameter Optimization with Epoch-Aware Trade-Offscited by 1
  18. 2025Multi-objective Hyperparameter Optimization in the Age of Deep Learningcited by 1
  19. 2025Center-Outward q-Dominance: A Sample-Computable Proxy for Strong Stochastic Dominance in Multi-Objective Optimisationcited by 1
  20. 2026Multi-Objective Multi-Fidelity Bayesian Optimization with Causal Priorscited by 1
  21. 2026Selecting Hyperparameters for Tree-Boostingcited by 1
  22. 2026BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Taskscited by 1
  23. 2026When to Use Which? Benchmarking Optimisers for Configurable Systems under Varying Budgetscited by 1
  24. 2022HPO: We won't get fooled againcited by 0
  25. 2023Iterative Deepening Hyperbandcited by 0
  26. 2024Strong convexity-guided hyper-parameter optimization for flatter lossescited by 0
  27. 2024Bootstrap Sampling Rate Greater than 1.0 May Improve Random Forest Performancecited by 0
  28. 2025POCAII: Parameter Optimization with Conscious Allocation using Iterative Intelligencecited by 0
  29. 2025VirnyFlow: Optimizing ML Pipelines for Accuracy, Fairness, and Stability at Scalecited by 0
  30. 2025Direct Regret Optimization in Bayesian Optimizationcited by 0
  31. 2025In-Context Decision Making for Optimizing Complex AutoML Pipelinescited by 0
  32. 2025Enhancing Performance and Calibration in Quantile Hyperparameter Optimizationcited by 0
  33. 2025Budget Allocation for Unknown Value Functions in a Lipschitz Spacecited by 0
  34. 2025Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Networkcited by 0
  35. 2026Dynamic Hyperparameter Importance for Efficient Multi-Objective Optimizationcited by 0
  36. 2026Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spacescited by 0
  37. 2026SMOG: Scalable Meta-Learning for Multi-Objective Bayesian Optimizationcited by 0
  38. 2026mlr3mbo: Bayesian Optimization in Rcited by 0

50 most cited of 61 All 61 (JSON)