mlsanityData to 7 Oct 2026

Deep Learning for Bayesian Optimization of Scientific Problems with High-Dimensional Structure

arXiv 2104.11667 ↗First posted 23 Apr 2021cs.LG physics.app-ph physics.chem-phcited by 15

Not on the map. Why?

Abstract (arXiv)

Bayesian optimization (BO) is a popular paradigm for global optimization of expensive black-box functions, but there are many domains where the function is not completely a black-box. The data may have some known structure (e.g. symmetries) and/or the data generation process may be a composite process that yields useful intermediate or auxiliary information in addition to the value of the optimization objective. However, surrogate models traditionally employed in BO, such as Gaussian Processes (GPs), scale poorly with dataset size and do not easily accommodate known structure. Instead, we use Bayesian neural networks, a class of scalable and flexible surrogate models with inductive biases, to extend BO to complex, structured problems with high dimensionality. We demonstrate BO on a number of realistic problems in physics and chemistry, including topology optimization of photonic crystal materials using convolutional neural networks, and chemical property optimization of molecules using graph neural networks. On these complex tasks, we show that neural networks often outperform GPs as surrogate models for BO in terms of both sampling efficiency and computational cost.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 55 of its 103 references that are in the map. First posted on arXiv 23 Apr 2021.

Builds on

55 of 103 references.

References · 55

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  6. 2017Deep Bayesian Active Learning with Image Datacited by 812
  7. 2016Automatic chemical design using a data-driven continuous representation of moleculescited by 762
  8. 2018Junction Tree Variational Autoencoder for Molecular Graph Generationcited by 647
  9. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  10. 2015Weight Uncertainty in Neural Networkscited by 554
  11. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  12. 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
38 more
  1. 2015Deep Kernel Learningcited by 440
  2. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  3. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  4. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  5. 2017Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equationscited by 362
  6. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 359
  7. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  8. 2020Discovering Symbolic Models from Deep Learning with Inductive Biasescited by 335
  9. 2017Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphscited by 307
  10. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  11. 2017Grammar Variational Autoencodercited by 283
  12. 2018When Gaussian Process Meets Big Data: A Review of Scalable GPscited by 205
  13. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  14. 2019Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Visioncited by 182
  15. 2020Training independent subnetworks for robust predictioncited by 177
  16. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
  17. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  18. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  19. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 146
  20. 2018Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problemscited by 144
  21. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 131
  22. 2017Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modelingcited by 130
  23. 2019Functional Variational Bayesian Neural Networkscited by 127
  24. 2020Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retrainingcited by 121
  25. 2019Exact Gaussian Processes on a Million Data Pointscited by 118
  26. 2017Batched Large-scale Bayesian Optimization in High-dimensional Spacescited by 116
  27. 2018Maximizing acquisition functions for Bayesian optimizationcited by 113
  28. 2020LieTransformer: Equivariant self-attention for Lie Groupscited by 100
  29. 2018Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Studycited by 90
  30. 2020Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Searchcited by 87
  31. 2014Freeze-Thaw Bayesian Optimizationcited by 86
  32. 2019ChemBO: Bayesian Optimization of Small Organic Molecules with Synthesizable Recommendationscited by 77
  33. 2019DScribe: Library of Descriptors for Machine Learning in Materials Sciencecited by 63
  34. 2020Graph Neural Networks in TensorFlow and Keras with Spektralcited by 59
  35. 2019Bayesian Optimization of Composite Functionscited by 45
  36. 2017Convolutional Gaussian Processescited by 44
  37. 2017Global optimization of Lipschitz functionscited by 39
  38. 2019Scalable Exact Inference in Multi-Output Gaussian Processescited by 25

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 3; 2026: 2.

15 papers.

Sort:

By area:

  1. 2023GFlowNets for AI-Driven Scientific Discoverycited by 63
  2. 2023A Study of Bayesian Neural Network Surrogates for Bayesian Optimizationcited by 50
  3. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 46
  4. 2024Bayesian Optimization via Continual Variational Last Layer Trainingcited by 13
  5. 2024Composite Bayesian Optimization In Function Spaces Using NEON -- Neural Epistemic Operator Networkscited by 10
  6. 2024Bayesian optimized deep ensemble for uncertainty quantification of deep neural networks: a system safety case study on sodium fast reactor thermal stratification modelingcited by 5
  7. 2023Neural-BO: A Black-box Optimization Algorithm using Deep Neural Networkscited by 2
  8. 2023Datasets and Benchmarks for Nanophotonic Structure and Parametric Design Simulationscited by 1
  9. 2025Bayesian Optimization with Preference Exploration using a Monotonic Neural Network Ensemblecited by 1
  10. 2026Multimodal Scientific Learning Beyond Diffusions and Flowscited by 1
  11. 2024Beyond Regrets: Geometric Metrics for Bayesian Optimizationcited by 0
  12. 2024PINN-BO: A Black-box Optimization Algorithm using Physics-Informed Neural Networkscited by 0
3 more
  1. 2025Direct Regret Optimization in Bayesian Optimizationcited by 0
  2. 2025Efficient Sliced Wasserstein Distance Computation via Adaptive Bayesian Optimizationcited by 0
  3. 2026Goal-Oriented Influence-Maximizing Data Acquisition for Learning and Optimizationcited by 0