# In-Context Learning for Latent Space Bayesian Optimization

arXiv: https://arxiv.org/abs/2606.09664 · HTML: https://mlsanity.com/paper/2606.09664/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 30 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Builds on (most cited first, 26 of 26)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [Protein Design with Guided Discrete Diffusion](https://mlsanity.com/paper/2305.20009/) (2023)
- [TabICLv2: A better, faster, scalable, and open tabular foundation model](https://mlsanity.com/paper/2602.11139/) (2026)
- [Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining](https://mlsanity.com/paper/2006.09191/) (2020)
- [Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoders](https://mlsanity.com/paper/2203.12742/) (2022)
- [Local Latent Space Bayesian Optimization over Structured Inputs](https://mlsanity.com/paper/2201.11872/) (2022)
- [PFNs4BO: In-Context Learning for Bayesian Optimization](https://mlsanity.com/paper/2305.17535/) (2023)
- [Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data](https://mlsanity.com/paper/2507.03971/) (2025)
- [End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes](https://mlsanity.com/paper/2305.15930/) (2023)
- [Position: The Future of Bayesian Prediction Is Prior-Fitted](https://mlsanity.com/paper/2505.23947/) (2025)
- [Latent Bayesian Optimization via Autoregressive Normalizing Flows](https://mlsanity.com/paper/2504.14889/) (2025)
- [Advancing Bayesian Optimization via Learning Correlated Latent Space](https://mlsanity.com/paper/2310.20258/) (2023)
- [TabPFN: One Model to Rule Them All?](https://mlsanity.com/paper/2505.20003/) (2025)
- [Return of the Latent Space COWBOYS: Re-thinking the use of VAEs for Bayesian Optimisation of Structured Spaces](https://mlsanity.com/paper/2507.03910/) (2025)
- [Inversion-based Latent Bayesian Optimization](https://mlsanity.com/paper/2411.05330/) (2024)
- [PABBO: Preferential Amortized Black-Box Optimization](https://mlsanity.com/paper/2503.00924/) (2025)
- [In-Context Multi-Objective Optimization](https://mlsanity.com/paper/2512.11114/) (2025)
- [GIT-BO: High-Dimensional Bayesian Optimization with Tabular Foundation Models](https://mlsanity.com/paper/2505.20685/) (2025)
- [None To Optima in Few Shots: Bayesian Optimization with MDP Priors](https://mlsanity.com/paper/2511.01006/) (2025)
- [In-Context Black-Box Optimization with Unreliable Feedback](https://mlsanity.com/paper/2605.06187/) (2026)
- [Time-Aware Latent Space Bayesian Optimization](https://mlsanity.com/paper/2603.00935/) (2026)

## Abstract (arXiv)

Bayesian optimization (BO) is a central tool for sample-efficient design, and latent-space Bayesian optimization (LSBO) extends it to structured objects such as molecules and proteins. In parallel, tabular foundation models such as TabPFN and TabICL now achieve state-of-the-art regression performance and are increasingly used as BO surrogates. Because their Bayesian behavior is induced by large synthetic pretraining collections, the composition of this pretraining distribution is crucial. LSBO creates a distinctive mismatch: the induced map from latent code to objective value differs markedly from the regression tasks used to train current in-context models. We address this mismatch by complementing the pretraining stage of tabular foundation model surrogates with synthetic optimization tasks defined on the latent space of a molecular VAE. The continued-pretraining objective features a regularizer that anchors the model to the original checkpoint, preserving its broad regression prior while avoiding overspecialization to the adaptation tasks. On held-out molecular optimization benchmarks, the resulting model achieves strong performance, supporting the relevance of LSBO-specific adaptation for in-context surrogates.