mlsanityData to 7 Oct 2026

Understanding High-Dimensional Bayesian Optimization

arXiv 2502.09198 ↗First posted 13 Feb 2025cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Recent work reported that simple Bayesian optimization (BO) methods perform well for high-dimensional real-world tasks, seemingly contradicting prior work and tribal knowledge. This paper investigates why. We identify underlying challenges that arise in high-dimensional BO and explain why recent methods succeed. Our empirical analysis shows that vanishing gradients caused by Gaussian process (GP) initialization schemes play a major role in the failures of high-dimensional Bayesian optimization (HDBO) and that methods that promote local search behaviors are better suited for the task. We find that maximum likelihood estimation (MLE) of GP length scales suffices for state-of-the-art performance. Based on this, we propose a simple variant of MLE called MSR that leverages these findings to achieve state-of-the-art performance on a comprehensive set of real-world applications. We present targeted experiments to illustrate and confirm our findings.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 31 of its 56 references that are in the map. First posted on arXiv 13 Feb 2025.

Builds on

31 of 56 references.

References · 31

  1. 2018A Tutorial on Bayesian Optimizationcited by 734
  2. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 359
  3. 2013Bayesian Optimization in a Billion Dimensions via Random Embeddingscited by 178
  4. 2023Unexpected Improvements to Expected Improvement for Bayesian Optimizationcited by 144
  5. 2021High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspacescited by 139
  6. 2015High Dimensional Bayesian Optimisation and Bandits via Additive Modelscited by 137
  7. 2020Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retrainingcited by 121
  8. 2017Batched Large-scale Bayesian Optimization in High-dimensional Spacescited by 116
  9. 2020Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimizationcited by 89
  10. 2020Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Searchcited by 87
  11. 2024Vanilla Bayesian Optimization Performs Great in High Dimensionscited by 87
  12. 2019Combinatorial Bayesian Optimization using the Graph Cartesian Productcited by 78
19 more
  1. 2022Local Latent Space Bayesian Optimization over Structured Inputscited by 78
  2. 2021A survey on high-dimensional Gaussian process modeling with application to Bayesian optimizationcited by 76
  3. 2023Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspacescited by 50
  4. 2021LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lassocited by 41
  5. 2022Monte Carlo Tree Search based Variable Selection for High Dimensional Bayesian Optimizationcited by 37
  6. 2023Are Random Decompositions all we need in High Dimensional Bayesian Optimisation?cited by 35
  7. 2024Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimizationcited by 33
  8. 2023Bounce: Reliable High-Dimensional Bayesian Optimization for Combinatorial and Mixed Spacescited by 27
  9. 2023Bayesian Optimization over High-Dimensional Combinatorial Spaces via Dictionary-based Embeddingscited by 24
  10. 2017Decentralized High-Dimensional Bayesian Optimization with Factor Graphscited by 23
  11. 2020High-Dimensional Bayesian Optimization via Tree-Structured Additive Modelscited by 22
  12. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  13. 2017Bayesian Optimization Using Domain Knowledge on the ATRIAS Bipedcited by 17
  14. 2022Maximum Likelihood Estimation in Gaussian Process Regression is Ill-Posedcited by 16
  15. 2020Semi-supervised Embedding Learning for High-dimensional Bayesian Optimizationcited by 11
  16. 2022Skill-based Multi-objective Reinforcement Learning of Industrial Robot Tasks with Planning and Knowledge Integrationcited by 10
  17. 2023Relaxing the Additivity Constraints in Decentralized No-Regret High-Dimensional Bayesian Optimizationcited by 8
  18. 2025Exploring Exploration in Bayesian Optimizationcited by 8
  19. 2023High-dimensional Bayesian Optimization with Group Testingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 14.

19 papers.

Sort:

By area:

  1. 2025Large language models as uncertainty-calibrated optimizers for experimental discoverycited by 10
  2. 2025We Still Don't Understand High-Dimensional Bayesian Optimizationcited by 9
  3. 2025NeST-BO: Fast Local Bayesian Optimization via Newton-Step Targeting of Gradient and Hessian Informationcited by 6
  4. 2026BayeSQP: Bayesian Optimization through Sequential Quadratic Programmingcited by 5
  5. 2025TextBO: Bayesian Optimization in Language Space for Eval-Efficient Self-Improving AIcited by 3
  6. 2026Do We Really Need to Approach the Entire Pareto Front in Many-Objective Bayesian Optimisation?cited by 2
  7. 2025Bencher: Simple and Reproducible Benchmarking for Black-Box Optimizationcited by 1
  8. 2026BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Taskscited by 1
  9. 2026Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimizationcited by 1
  10. 2026Gradient-Informed Bayesian and Interior Point Optimization for Efficient Inverse Design in Nanophotonicscited by 0
  11. 2026LAGO: A Local-Global Optimization Framework Combining Trust Region Methods and Bayesian Optimizationcited by 0
  12. 2026Trust Region Constrained Bayesian Optimization with Penalized Constraint Handlingcited by 0
7 more
  1. 2026Rethinking Trust Region Bayesian Optimization in High Dimensionscited by 0
  2. 2026Proximal basin hopping: global optimization with guaranteescited by 0
  3. 2026GRAPE: Gradient Refinement and Progress-Aware Exploitation for Query-Efficient High-Dimensional Bayesian Optimizationcited by 0
  4. 2026Converge Then Diversify: Decoupling Convergence and Diversity in Multi-Objective Bayesian Optimisationcited by 0
  5. 2026MF-SCBO : Multi-fidelity Scalable Constrained Bayesian Optimizationcited by 0
  6. 2026Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methodscited by 0
  7. 2026Agentic High-Dimensional Bayesian Optimization with Hypothesis- and Evidence-Guided Searchcited by 0