# Bayesian Optimization with High-Dimensional Outputs

arXiv: https://arxiv.org/abs/2106.12997 · HTML: https://mlsanity.com/paper/2106.12997/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 64 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 22 of 22)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2006.05078/) (2020)
- [Efficiently Sampling Functions from Gaussian Process Posteriors](https://mlsanity.com/paper/2002.09309/) (2020)
- [Exact Gaussian Processes on a Million Data Points](https://mlsanity.com/paper/1903.08114/) (2019)
- [Maximizing acquisition functions for Bayesian optimization](https://mlsanity.com/paper/1805.10196/) (2018)
- [Bayesian Optimization with Unknown Constraints](https://mlsanity.com/paper/1403.5607/) (2014)
- [Scalable Constrained Bayesian Optimization](https://mlsanity.com/paper/2002.08526/) (2020)
- [Pathwise Conditioning of Gaussian Processes](https://mlsanity.com/paper/2011.04026/) (2020)
- [Constant-Time Predictive Distributions for Gaussian Processes](https://mlsanity.com/paper/1803.06058/) (2018)
- [Bayesian Optimization of Composite Functions](https://mlsanity.com/paper/1906.01537/) (2019)
- [Efficient Nonmyopic Bayesian Optimization via One-Shot Multi-Step Trees](https://mlsanity.com/paper/2006.15779/) (2020)
- [Bayesian Optimization of Risk Measures](https://mlsanity.com/paper/2007.05554/) (2020)
- [Fast Matrix Square Roots with Applications to Gaussian Processes and Bayesian Optimization](https://mlsanity.com/paper/2006.11267/) (2020)
- [Scalable Exact Inference in Multi-Output Gaussian Processes](https://mlsanity.com/paper/1911.06287/) (2019)
- [Optimizing Coverage and Capacity in Cellular Networks using Machine Learning](https://mlsanity.com/paper/2010.13710/) (2020)
- [No-regret Algorithms for Multi-task Bayesian Optimization](https://mlsanity.com/paper/2008.08885/) (2020)
- [Efficient Modeling of Latent Information in Supervised Learning using Gaussian Processes](https://mlsanity.com/paper/1705.09862/) (2017)
- [Interferobot: aligning an optical interferometer by a reinforcement learning agent](https://mlsanity.com/paper/2006.02252/) (2020)
- [Quantifying the Effects of Contact Tracing, Testing, and Containment Measures in the Presence of Infection Hotspots](https://mlsanity.com/paper/2004.07641/) (2020)
- [Non-linear Multitask Learning with Deep Gaussian Processes](https://mlsanity.com/paper/1905.12407/) (2019)

## Built on by (most cited first, 30 of 32)

- [Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoders](https://mlsanity.com/paper/2203.12742/) (2022)
- [Multi-Objective GFlowNets](https://mlsanity.com/paper/2210.12765/) (2022)
- [Recent Advances in Bayesian Optimization](https://mlsanity.com/paper/2206.03301/) (2022)
- [A Study of Bayesian Neural Network Surrogates for Bayesian Optimization](https://mlsanity.com/paper/2305.20028/) (2023)
- [Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterization](https://mlsanity.com/paper/2210.10199/) (2022)
- [Thinking inside the box: A tutorial on grey-box Bayesian optimization](https://mlsanity.com/paper/2201.00272/) (2022)
- [A General Recipe for Likelihood-free Bayesian Optimization](https://mlsanity.com/paper/2206.13035/) (2022)
- [Composite Bayesian Optimization In Function Spaces Using NEON -- Neural Epistemic Operator Networks](https://mlsanity.com/paper/2404.03099/) (2024)
- [Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure](https://mlsanity.com/paper/2410.09239/) (2024)
- [Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features](https://mlsanity.com/paper/2407.00584/) (2024)
- [Optimizing High-Dimensional Physics Simulations via Composite Bayesian Optimization](https://mlsanity.com/paper/2111.14911/) (2021)
- [No-Regret Constrained Bayesian Optimization of Noisy and Expensive Hybrid Models using Differentiable Quantile Function Approximations](https://mlsanity.com/paper/2305.03824/) (2023)
- [ArchGym: An Open-Source Gymnasium for Machine Learning Assisted Architecture Design](https://mlsanity.com/paper/2306.08888/) (2023)
- [Joint Composite Latent Space Bayesian Optimization](https://mlsanity.com/paper/2311.02213/) (2023)
- [Scalable Gaussian Processes with Latent Kronecker Structure](https://mlsanity.com/paper/2506.06895/) (2025)
- [Bayesian Optimization for Function Compositions with Applications to Dynamic Pricing](https://mlsanity.com/paper/2303.11954/) (2023)
- [Multi-Fidelity Bayesian Optimization With Across-Task Transferable Max-Value Entropy Search](https://mlsanity.com/paper/2403.09570/) (2024)
- [Knowledge-aware Evolutionary Graph Neural Architecture Search](https://mlsanity.com/paper/2411.17339/) (2024)
- [Sampling from Gaussian Processes: A Tutorial and Applications in Global Sensitivity Analysis and Optimization](https://mlsanity.com/paper/2507.14746/) (2025)
- [Multi-armed bandits for resource efficient, online optimization of language model pre-training: the use case of dynamic masking](https://mlsanity.com/paper/2203.13151/) (2022)
- [Scalable Bayesian optimization with high-dimensional outputs using randomized prior networks](https://mlsanity.com/paper/2302.07260/) (2023)
- [BEACON: A Bayesian Optimization Inspired Strategy for Efficient Novelty Search](https://mlsanity.com/paper/2406.03616/) (2024)
- [Multi-Output Robust and Conjugate Gaussian Processes](https://mlsanity.com/paper/2510.26401/) (2025)
- [Volatility Based Kernels and Moving Average Means for Accurate Forecasting with Gaussian Processes](https://mlsanity.com/paper/2207.06544/) (2022)
- [High-Dimensional Bayesian Optimisation with Large-Scale Constraints -- An Application to Aeroelastic Tailoring](https://mlsanity.com/paper/2312.08891/) (2023)
- [Derivation of Output Correlation Inferences for Multi-Output (aka Multi-Task) Gaussian Process](https://mlsanity.com/paper/2501.07964/) (2025)
- [Multi-fidelity Bayesian Data-Driven Design of Energy Absorbing Spinodoid Cellular Structures](https://mlsanity.com/paper/2507.22079/) (2025)
- [Gaussian Processes Sampling with Sparse Grids under Additive Schwarz Preconditioner](https://mlsanity.com/paper/2408.00206/) (2024)
- [Respecting the limit:Bayesian optimization with a bound on the optimal value](https://mlsanity.com/paper/2411.04744/) (2024)
- [Multivariate and Online Transfer Learning with Uncertainty Quantification](https://mlsanity.com/paper/2411.12555/) (2024)

## Abstract (arXiv)

Bayesian Optimization is a sample-efficient black-box optimization procedure that is typically applied to problems with a small number of independent objectives. However, in practice we often wish to optimize objectives defined over many correlated outcomes (or "tasks"). For example, scientists may want to optimize the coverage of a cell tower network across a dense grid of locations. Similarly, engineers may seek to balance the performance of a robot across dozens of different environments via constrained or robust optimization. However, the Gaussian Process (GP) models typically used as probabilistic surrogates for multi-task Bayesian Optimization scale poorly with the number of outcomes, greatly limiting applicability. We devise an efficient technique for exact multi-task GP sampling that combines exploiting Kronecker structure in the covariance matrices with Matheron's identity, allowing us to perform Bayesian Optimization using exact multi-task GP models with tens of thousands of correlated outputs. In doing so, we achieve substantial improvements in sample efficiency compared to existing approaches that only model aggregate functions of the outcomes. We demonstrate how this unlocks a new class of applications for Bayesian Optimization across a range of tasks in science and engineering, including optimizing interference patterns of an optical interferometer with more than 65,000 outputs.