mlsanityData to 7 Oct 2026

Vanilla Bayesian Optimization Performs Great in High Dimensions

arXiv 2402.02229 ↗First posted 3 Feb 2024cs.LG stat.MLcited by 87

Not on the map. Why?

Abstract (arXiv)

High-dimensional problems have long been considered the Achilles' heel of Bayesian optimization algorithms. Spurred by the curse of dimensionality, a large collection of algorithms aim to make it more performant in this setting, commonly by imposing various simplifying assumptions on the objective. In this paper, we identify the degeneracies that make vanilla Bayesian optimization poorly suited to high-dimensional tasks, and further show how existing algorithms address these degeneracies through the lens of lowering the model complexity. Moreover, we propose an enhancement to the prior assumptions that are typical to vanilla Bayesian optimization algorithms, which reduces the complexity to manageable levels without imposing structural restrictions on the objective. Our modification - a simple scaling of the Gaussian process lengthscale prior with the dimensionality - reveals that standard Bayesian optimization works drastically better than previously thought in high dimensions, clearly outperforming existing state-of-the-art algorithms on multiple commonly considered real-world high-dimensional tasks.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 40 of its 69 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

40 of 69 references.

References · 40

  1. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
  2. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 359
  3. 2013Bayesian Optimization in a Billion Dimensions via Random Embeddingscited by 178
  4. 2023Unexpected Improvements to Expected Improvement for Bayesian Optimizationcited by 144
  5. 2021High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspacescited by 139
  6. 2015High Dimensional Bayesian Optimisation and Bandits via Additive Modelscited by 137
  7. 2020Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimizationcited by 89
  8. 2020Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Searchcited by 87
  9. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  10. 2021The Promises and Pitfalls of Deep Kernel Learningcited by 81
  11. 2022Local Latent Space Bayesian Optimization over Structured Inputscited by 78
  12. 2021A survey on high-dimensional Gaussian process modeling with application to Bayesian optimizationcited by 76
28 more
  1. 2019Adaptive and Safe Bayesian Optimization in High Dimensions via One-Dimensional Subspacescited by 73
  2. 2014Input Warping for Bayesian Optimization of Non-stationary Functionscited by 66
  3. 2018Bayesian Optimization of Combinatorial Structurescited by 64
  4. 2018High-Dimensional Bayesian Optimization via Additive Models with Overlapping Groupscited by 58
  5. 2017Batched High-dimensional Bayesian Optimization via Structural Kernel Learningcited by 51
  6. 2021Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spacescited by 51
  7. 2019No-Regret Bayesian Optimization with Unknown Hyperparameterscited by 50
  8. 2023Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspacescited by 50
  9. 2021LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lassocited by 41
  10. 2018High Dimensional Bayesian Optimization Using Dropoutcited by 40
  11. 2021Local policy search with Bayesian optimizationcited by 37
  12. 2022Monte Carlo Tree Search based Variable Selection for High Dimensional Bayesian Optimizationcited by 37
  13. 2023Are Random Decompositions all we need in High Dimensional Bayesian Optimisation?cited by 35
  14. 2013Active Learning of Linear Embeddings for Gaussian Processescited by 34
  15. 2020Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriorscited by 29
  16. 2022Local Bayesian optimization via maximizing probability of descentcited by 29
  17. 2017On the choice of the low-dimensional domain for global optimization via random embeddingscited by 25
  18. 2021Meta-Learning Reliable Priors in the Function Spacecited by 23
  19. 2020High-Dimensional Bayesian Optimization via Tree-Structured Additive Modelscited by 22
  20. 2023Self-Correcting Bayesian Optimization through Bayesian Active Learningcited by 22
  21. 2017Constrained Bayesian Optimization for Automatic Chemical Designcited by 20
  22. 2023The Behavior and Convergence of Local Bayesian Optimizationcited by 19
  23. 2014Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spacescited by 16
  24. 2022Maximum Likelihood Estimation in Gaussian Process Regression is Ill-Posedcited by 16
  25. 2017Correcting boundary over-exploration deficiencies in Bayesian optimization with virtual derivative sign observationscited by 11
  26. 2020What do you Mean? The Role of the Mean Function in Bayesian Optimisationcited by 7
  27. 2016Fast methods for training Gaussian processes on large data setscited by 5
  28. 2022Bayesian Optimization with Informative Covariancecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 35; 2026: 42.

87 papers.

Sort:

By area:

  1. 2024Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimizationcited by 33
  2. 2024A survey and benchmark of high-dimensional Bayesian optimization of discrete sequencescited by 22
  3. 2025Understanding High-Dimensional Bayesian Optimizationcited by 19
  4. 2025Return of the Latent Space COWBOYS: Re-thinking the use of VAEs for Bayesian Optimisation of Structured Spacescited by 13
  5. 2024Generative Adversarial Model-Based Optimization via Source Critic Regularizationcited by 10
  6. 2024Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structurecited by 9
  7. 2024Hyperband-based Bayesian Optimization for Black-box Prompt Selectioncited by 9
  8. 2025We Still Don't Understand High-Dimensional Bayesian Optimizationcited by 9
  9. 2025Exploring Exploration in Bayesian Optimizationcited by 8
  10. 2024Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Featurescited by 7
  11. 2025Fast Direct: Query-Efficient Online Black-box Guidance for Diffusion-model Target Generationcited by 7
  12. 2025Koopman-Equivariant Gaussian Processescited by 6
38 more
  1. 2025NeST-BO: Fast Local Bayesian Optimization via Newton-Step Targeting of Gradient and Hessian Informationcited by 6
  2. 2026Adaptive Acquisition Selection for Bayesian Optimization with Large Language Modelscited by 6
  3. 2026BayeSQP: Bayesian Optimization through Sequential Quadratic Programmingcited by 5
  4. 2025Martingale Posterior Neural Networks for Fast Sequential Decision Makingcited by 4
  5. 2025Cooperative Design Optimization through Natural Language Interactioncited by 4
  6. 2025BBOPlace-Bench: Benchmarking Black-Box Optimization for Chip Placementcited by 4
  7. 2026Tighter Regret Lower Bound for Gaussian Process Bandits with Squared Exponential Kernel in Hyperspherecited by 4
  8. 2026LABO: LLM-Accelerated Bayesian Optimization through Broad Exploration and Selective Experimentationcited by 4
  9. 2024Regional Expected Improvement for Efficient Trust Region Selection in High-Dimensional Bayesian Optimizationcited by 3
  10. 2025A Unified Framework for Entropy Search and Expected Improvement in Bayesian Optimizationcited by 3
  11. 2025High Dimensional Bayesian Optimization using Lasso Variable Selectioncited by 3
  12. 2025The Catechol Benchmark: Time-series Solvent Selection Data for Few-shot Machine Learningcited by 3
  13. 2026Conditioning Gaussian Processes on Almost Anythingcited by 3
  14. 2024CATBench: A Compiler Autotuning Benchmarking Suite for Black-box Optimizationcited by 2
  15. 2025Learning Hyperparameters via a Data-Emphasized Variational Objectivecited by 2
  16. 2025BARK: A Fully Bayesian Tree Kernel for Black-box Optimizationcited by 2
  17. 2025Enhancing Trust-Region Bayesian Optimization via Newton Methodscited by 2
  18. 2025Fine-tuning LLMs with variational Bayesian last layer for high-dimensional Bayesian optimizationcited by 2
  19. 2025Informed Initialization for Bayesian Optimization and Active Learningcited by 2
  20. 2026Do We Really Need to Approach the Entire Pareto Front in Many-Objective Bayesian Optimisation?cited by 2
  21. 2024Dimensionality Reduction Techniques for Global Bayesian Optimisationcited by 1
  22. 2025Leveraging Axis-Aligned Subspaces for High-Dimensional Bayesian Optimization with Group Testingcited by 1
  23. 2025FigBO: A Generalized Acquisition Function Framework with Look-Ahead Capability for Bayesian Optimizationcited by 1
  24. 2025Bencher: Simple and Reproducible Benchmarking for Black-Box Optimizationcited by 1
  25. 2025A Guide to Bayesian Optimization in Bioprocess Engineeringcited by 1
  26. 2025BO4Mob: Bayesian Optimization Benchmarks for High-Dimensional Urban Mobility Problemcited by 1
  27. 2025Local Entropy Search over Descent Sequences for Bayesian Optimizationcited by 1
  28. 2026Adaptive Candidate Point Thompson Sampling for High-Dimensional Bayesian Optimizationcited by 1
  29. 2026BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Taskscited by 1
  30. 2026Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimizationcited by 1
  31. 2026α-PFN: Fast Entropy Search via In-Context Learningcited by 1
  32. 2026Search at the Cost of Sampling: Nearly-Instant Latent Space Bayesian Optimizationcited by 1
  33. 2024Sample-Efficient Bayesian Optimization with Transfer Learning for Heterogeneous Search Spacescited by 0
  34. 2025Covering Multiple Objectives with a Small Set of Solutions Using Bayesian Optimizationcited by 0
  35. 2025Differentially Private Hyperparameter Tuning using Local Bayesian Optimizationcited by 0
  36. 2025Bayesian Optimization over Bounded Domains with the Beta Product Kernelcited by 0
  37. 2025Scalable neural network-based blackbox optimizationcited by 0
  38. 2025Efficient Sliced Wasserstein Distance Computation via Adaptive Bayesian Optimizationcited by 0

50 most cited of 87 All 87 (JSON)