# Nonlinear Dimensionality Reduction Techniques for Bayesian Optimization

arXiv: https://arxiv.org/abs/2510.15435 · HTML: https://mlsanity.com/paper/2510.15435/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 56 references that are in the map.
- First posted on arXiv 17 Oct 2025.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [High Dimensional Bayesian Optimisation and Bandits via Additive Models](https://mlsanity.com/paper/1503.01673/) (2015)
- [Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining](https://mlsanity.com/paper/2006.09191/) (2020)
- [Vanilla Bayesian Optimization Performs Great in High Dimensions](https://mlsanity.com/paper/2402.02229/) (2024)
- [High-Dimensional Bayesian Optimization via Additive Models with Overlapping Groups](https://mlsanity.com/paper/1802.07028/) (2018)
- [High-Dimensional Bayesian Optimisation with Variational Autoencoders and Deep Metric Learning](https://mlsanity.com/paper/2106.03609/) (2021)
- [A Domain-Shrinking based Bayesian Optimization Algorithm with Order-Optimal Regret Performance](https://mlsanity.com/paper/2010.13997/) (2020)
- [Active Learning of Linear Embeddings for Gaussian Processes](https://mlsanity.com/paper/1310.6740/) (2013)
- [On the choice of the low-dimensional domain for global optimization via random embeddings](https://mlsanity.com/paper/1704.05318/) (2017)
- [Good practices for Bayesian Optimization of high dimensional structured spaces](https://mlsanity.com/paper/2012.15471/) (2020)
- [TVAE: Triplet-Based Variational Autoencoder using Metric Learning](https://mlsanity.com/paper/1802.04403/) (2018)
- [A warped kernel improving robustness in Bayesian optimization via random embeddings](https://mlsanity.com/paper/1411.3685/) (2014)
- [Statistical Guarantees for Variational Autoencoders using PAC-Bayesian Theory](https://mlsanity.com/paper/2310.04935/) (2023)
- [A dimensionality reduction technique for unconstrained global optimization of functions with low effective dimensionality](https://mlsanity.com/paper/2003.09673/) (2020)
- [Learning Non-Parametric Basis Independent Models from Point Queries via Low-Rank Methods](https://mlsanity.com/paper/1310.1826/) (2013)
- [Global optimization using random embeddings](https://mlsanity.com/paper/2107.12102/) (2021)
- [Escaping local minima with derivative-free methods: a numerical investigation](https://mlsanity.com/paper/1812.11343/) (2018)
- [Dimensionality Reduction Techniques for Global Bayesian Optimisation](https://mlsanity.com/paper/2412.09183/) (2024)

## Abstract (arXiv)

Bayesian optimisation (BO) enables sample-efficient global optimisation of expensive black-box functions but remains challenging in high dimensions. We investigate nonlinear dimensionality reduction to a sequence of low-dimensional latent-space BO (LSBO) problems. Early LSBO used linear random and supervised embeddings; building on Grosnit et al., we employ variational autoencoders (VAEs), deep metric loss for structured latent manifolds, and retraining to adapt the encoder-decoder pair to newly sampled regions. We couple LSBO with sequential domain reduction (SDR) directly in latent space (SDR-LSBO), narrowing search domains as evidence accumulates. Implemented in GPU-accelerated BoTorch with Mat'ern-5/2 Gaussian-process surrogates, our methods improve benchmark optimisation quality, and retraining can enhance BO performance. Comparisons with adaptive supervised linear random embeddings demonstrate the effectiveness of VAE-based BO for nonlinear low-dimensional structures. We analyse BO-VAE with a fixed pretrained representation, decomposing ambient-space simple regret into latent BO error and a fixed VAE-induced representation gap. Under a PAC-Bayes-certified reconstruction condition and standard fixed-prior assumptions for expected improvement with a Mat'ern-5/2 kernel, latent BO error vanishes as the evaluation budget increases, whereas the representation gap remains fixed and may impose a non-vanishing error floor. Visualisations empirically assess accessibility of the ambient optimum through the learned decoder. To our knowledge, this is the first study combining SDR with VAE-based LSBO. Our analysis clarifies metric shaping and retraining choices critical for scalable latent-space BO. For reproducibility, source code is available at https://github.com/L-Lok/Nonlinear-Dimensionality-Reduction-Techniques-for-Bayesian-Optimization.git.