# Deep Learning for Bayesian Optimization of Scientific Problems with High-Dimensional Structure

arXiv: https://arxiv.org/abs/2104.11667 · HTML: https://mlsanity.com/paper/2104.11667/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 103 references that are in the map.
- First posted on arXiv 23 Apr 2021.

## Builds on (most cited first, 30 of 55)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1711.10561/) (2017)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [Discovering Symbolic Models from Deep Learning with Inductive Biases](https://mlsanity.com/paper/2006.11287/) (2020)
- [Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs](https://mlsanity.com/paper/1704.02901/) (2017)
- [Grammar Variational Autoencoder](https://mlsanity.com/paper/1703.01925/) (2017)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [When Gaussian Process Meets Big Data: A Review of Scalable GPs](https://mlsanity.com/paper/1807.01065/) (2018)
- [Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning](https://mlsanity.com/paper/1902.03932/) (2019)
- [Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision](https://mlsanity.com/paper/1906.01620/) (2019)
- [Training independent subnetworks for robust prediction](https://mlsanity.com/paper/2010.06610/) (2020)
- [Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling](https://mlsanity.com/paper/1802.09127/) (2018)
- [Multiplicative Normalizing Flows for Variational Bayesian Neural Networks](https://mlsanity.com/paper/1703.01961/) (2017)
- [Neural Tangents: Fast and Easy Infinite Neural Networks in Python](https://mlsanity.com/paper/1912.02803/) (2019)

## Built on by (most cited first, 15 of 15)

- [GFlowNets for AI-Driven Scientific Discovery](https://mlsanity.com/paper/2302.00615/) (2023)
- [A Study of Bayesian Neural Network Surrogates for Bayesian Optimization](https://mlsanity.com/paper/2305.20028/) (2023)
- [A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?](https://mlsanity.com/paper/2402.05015/) (2024)
- [Bayesian Optimization via Continual Variational Last Layer Training](https://mlsanity.com/paper/2412.09477/) (2024)
- [Composite Bayesian Optimization In Function Spaces Using NEON -- Neural Epistemic Operator Networks](https://mlsanity.com/paper/2404.03099/) (2024)
- [Bayesian optimized deep ensemble for uncertainty quantification of deep neural networks: a system safety case study on sodium fast reactor thermal stratification modeling](https://mlsanity.com/paper/2412.08776/) (2024)
- [Neural-BO: A Black-box Optimization Algorithm using Deep Neural Networks](https://mlsanity.com/paper/2303.01682/) (2023)
- [Datasets and Benchmarks for Nanophotonic Structure and Parametric Design Simulations](https://mlsanity.com/paper/2310.19053/) (2023)
- [Bayesian Optimization with Preference Exploration using a Monotonic Neural Network Ensemble](https://mlsanity.com/paper/2501.18792/) (2025)
- [Multimodal Scientific Learning Beyond Diffusions and Flows](https://mlsanity.com/paper/2602.00960/) (2026)
- [Beyond Regrets: Geometric Metrics for Bayesian Optimization](https://mlsanity.com/paper/2401.01981/) (2024)
- [PINN-BO: A Black-box Optimization Algorithm using Physics-Informed Neural Networks](https://mlsanity.com/paper/2402.03243/) (2024)
- [Direct Regret Optimization in Bayesian Optimization](https://mlsanity.com/paper/2507.06529/) (2025)
- [Efficient Sliced Wasserstein Distance Computation via Adaptive Bayesian Optimization](https://mlsanity.com/paper/2509.17405/) (2025)
- [Goal-Oriented Influence-Maximizing Data Acquisition for Learning and Optimization](https://mlsanity.com/paper/2602.19578/) (2026)

## Abstract (arXiv)

Bayesian optimization (BO) is a popular paradigm for global optimization of expensive black-box functions, but there are many domains where the function is not completely a black-box. The data may have some known structure (e.g. symmetries) and/or the data generation process may be a composite process that yields useful intermediate or auxiliary information in addition to the value of the optimization objective. However, surrogate models traditionally employed in BO, such as Gaussian Processes (GPs), scale poorly with dataset size and do not easily accommodate known structure. Instead, we use Bayesian neural networks, a class of scalable and flexible surrogate models with inductive biases, to extend BO to complex, structured problems with high dimensionality. We demonstrate BO on a number of realistic problems in physics and chemistry, including topology optimization of photonic crystal materials using convolutional neural networks, and chemical property optimization of molecules using graph neural networks. On these complex tasks, we show that neural networks often outperform GPs as surrogate models for BO in terms of both sampling efficiency and computational cost.