mlsanityData to 7 Oct 2026

Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020

arXiv 2104.10201 ↗First posted 20 Apr 2021cs.LG cs.AI stat.MLcited by 146

Not on the map. Why?

Abstract (arXiv)

This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS 2020 which ran from July-October, 2020. The challenge emphasized the importance of evaluating derivative-free optimizers for tuning the hyperparameters of machine learning models. This was the first black-box optimization challenge with a machine learning emphasis. It was based on tuning (validation set) performance of standard machine learning models on real datasets. This competition has widespread impact as black-box optimization (e.g., Bayesian optimization) is relevant for hyperparameter tuning in almost every machine learning project as well as many applications outside of machine learning. The final leaderboard was determined using the optimization performance on held-out (hidden) objective functions, where the optimizers ran without human intervention. Baselines were set using the default settings of several open-source black-box optimization packages as well as random search.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). Builds on 24 of its 67 references that are in the map. First posted on arXiv 20 Apr 2021.

Builds on

24 of 67 references.

References · 24

  1. 2018A Tutorial on Bayesian Optimizationcited by 734
  2. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  3. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 359
  4. 2020Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimizationcited by 176
  5. 2016COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Settingcited by 135
  6. 2015Batch Bayesian Optimization via Local Penalizationcited by 103
  7. 2017Constrained Bayesian Optimization with Noisy Experimentscited by 103
  8. 2020Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Searchcited by 87
  9. 2020Scalable Constrained Bayesian Optimizationcited by 84
  10. 2016Multi-Information Source Optimizationcited by 83
  11. 2015Bayesian optimization for materials designcited by 78
  12. 2017Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Spacecited by 76
12 more
  1. 2016The Parallel Knowledge Gradient Method for Batch Bayesian Optimizationcited by 73
  2. 2019Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonflycited by 66
  3. 2015Predictive Entropy Search for Bayesian Optimization with Unknown Constraintscited by 51
  4. 2014An introduction to Docker for reproducible research, with examples from the R environmentcited by 25
  5. 2017GPflowOpt: A Bayesian Optimization Library using TensorFlowcited by 18
  6. 2020Solving Black-Box Optimization Challenge via Learning Search Space Partition for Local Bayesian Optimizationcited by 7
  7. 2019pySOT and POAP: An event-driven asynchronous framework for surrogate optimizationcited by 6
  8. 2020GPU Accelerated Exhaustive Search for Optimal Ensemble of Black-Box Optimization Algorithmscited by 5
  9. 2020Squirrel: A Switching Hyperparameter Optimizercited by 4
  10. 2019ProBO: Versatile Bayesian Optimization Using Any Probabilistic Programming Languagecited by 3
  11. 2020Adaptive Local Bayesian Optimization Over Multiple Discrete Variablescited by 3
  12. 2020Better call Surrogates: A hybrid Evolutionary Algorithm for Hyperparameter optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 36; 2023: 30; 2024: 26; 2025: 22; 2026: 13.

146 papers.

Sort:

By area:

  1. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  2. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  3. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 229
  4. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  5. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 185
  6. 2023Unexpected Improvements to Expected Improvement for Bayesian Optimizationcited by 144
  7. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  8. 2023Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performancecited by 143
  9. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  10. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 108
  11. 2022Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoderscited by 99
  12. 2021Multi-Objective Bayesian Optimization over High-Dimensional Search Spacescited by 87
38 more
  1. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  2. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  3. 2023PFNs4BO: In-Context Learning for Bayesian Optimizationcited by 70
  4. 2022πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimizationcited by 69
  5. 2021Pre-trained Gaussian Processes for Bayesian Optimizationcited by 63
  6. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 58
  7. 2021High-Dimensional Bayesian Optimisation with Variational Autoencoders and Deep Metric Learningcited by 54
  8. 2024LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generationcited by 45
  9. 2022GAUCHE: A Library for Gaussian Processes in Chemistrycited by 40
  10. 2022Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterizationcited by 35
  11. 2023Are Random Decompositions all we need in High Dimensional Bayesian Optimisation?cited by 35
  12. 2023MetaBox: A Benchmark Platform for Meta-Black-Box Optimization with Reinforcement Learningcited by 35
  13. 2022Discovering Many Diverse Solutions with Bayesian Optimizationcited by 29
  14. 2022Thinking inside the box: A tutorial on grey-box Bayesian optimizationcited by 26
  15. 2022Surrogate modeling for Bayesian optimization beyond a single Gaussian processcited by 22
  16. 2023Self-Correcting Bayesian Optimization through Bayesian Active Learningcited by 22
  17. 2024A survey and benchmark of high-dimensional Bayesian optimization of discrete sequencescited by 22
  18. 2021A Nonmyopic Approach to Cost-Constrained Bayesian Optimizationcited by 18
  19. 2023Comparison of High-Dimensional Bayesian Optimization Algorithms on BBOBcited by 18
  20. 2023Active Learning and Bayesian Optimization: a Unified Perspective to Learn with a Goalcited by 17
  21. 2024Evolve Cost-aware Acquisition Functions Using Large Language Modelscited by 17
  22. 2021Deep Learning for Bayesian Optimization of Scientific Problems with High-Dimensional Structurecited by 15
  23. 2021Latency-Aware Neural Architecture Search with Multi-Objective Bayesian Optimizationcited by 15
  24. 2022AntBO: Towards Real-World Automated Antibody Design with Combinatorial Bayesian Optimisationcited by 14
  25. 2024Reinforced In-Context Black-Box Optimizationcited by 14
  26. 2025OptunaHub: A Platform for Black-Box Optimizationcited by 14
  27. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  28. 2024ZipNN: Lossless Compression for AI Modelscited by 12
  29. 2025Posterior Inference with Diffusion Models for High-dimensional Black-box Optimizationcited by 12
  30. 2021Triangulation candidates for Bayesian optimizationcited by 11
  31. 2022Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classificationcited by 10
  32. 2025LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithmscited by 10
  33. 2022c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimizationcited by 9
  34. 2023Transfer Learning for Bayesian Optimization on Heterogeneous Search Spacescited by 9
  35. 2021Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiationcited by 7
  36. 2022Evaluating histopathology transfer learning with ChampKitcited by 7
  37. 2023On the role of Model Uncertainties in Bayesian Optimizationcited by 7
  38. 2024Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimizationcited by 7

50 most cited of 146 All 146 (JSON)