# EARL-BO: Reinforcement Learning for Multi-Step Lookahead, High-Dimensional Bayesian Optimization

arXiv: https://arxiv.org/abs/2411.00171 · HTML: https://mlsanity.com/paper/2411.00171/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 28 of its 49 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [Unexpected Improvements to Expected Improvement for Bayesian Optimization](https://mlsanity.com/paper/2310.20708/) (2023)
- [Convolutional Conditional Neural Processes](https://mlsanity.com/paper/1910.13556/) (2019)
- [High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces](https://mlsanity.com/paper/2103.00349/) (2021)
- [Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining](https://mlsanity.com/paper/2006.09191/) (2020)
- [Towards Learning Universal Hyperparameter Optimizers with Transformers](https://mlsanity.com/paper/2205.13320/) (2022)
- [Recent Advances in Bayesian Optimization](https://mlsanity.com/paper/2206.03301/) (2022)
- [HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML](https://mlsanity.com/paper/2106.06257/) (2021)
- [Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization](https://mlsanity.com/paper/1904.02642/) (2019)
- [Data-efficient Domain Randomization with Bayesian Optimization](https://mlsanity.com/paper/2003.02471/) (2020)
- [GLASSES: Relieving The Myopia Of Bayesian Optimisation](https://mlsanity.com/paper/1510.06299/) (2015)
- [End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes](https://mlsanity.com/paper/2305.15930/) (2023)
- [Decentralized High-Dimensional Bayesian Optimization with Factor Graphs](https://mlsanity.com/paper/1711.07033/) (2017)
- [Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization](https://mlsanity.com/paper/2106.04335/) (2021)
- [Efficient Rollout Strategies for Bayesian Optimization](https://mlsanity.com/paper/2002.10539/) (2020)
- [Bayesian optimization as a flexible and efficient design framework for sustainable process systems](https://mlsanity.com/paper/2401.16373/) (2024)
- [SnAKe: Bayesian Optimization with Pathwise Exploration](https://mlsanity.com/paper/2202.00060/) (2022)
- [Reinforced In-Context Black-Box Optimization](https://mlsanity.com/paper/2402.17423/) (2024)
- [Kernel Identification Through Transformers](https://mlsanity.com/paper/2106.08185/) (2021)
- [Pretrained Optimization Model for Zero-Shot Black Box Optimization](https://mlsanity.com/paper/2405.03728/) (2024)
- [Multi-Objective Constrained Optimization for Energy Applications via Tree Ensembles](https://mlsanity.com/paper/2111.03140/) (2021)
- [RTDK-BO: High Dimensional Bayesian Optimization with Reinforced Transformer Deep kernels](https://mlsanity.com/paper/2310.03912/) (2023)
- [PG-LBO: Enhancing High-Dimensional Bayesian Optimization with Pseudo-Label and Gaussian Process Guidance](https://mlsanity.com/paper/2312.16983/) (2023)
- [Global Optimization of Gaussian Process Acquisition Functions Using a Piecewise-Linear Kernel Approximation](https://mlsanity.com/paper/2410.16893/) (2024)
- [On Estimating the Gradient of the Expected Information Gain in Bayesian Experimental Design](https://mlsanity.com/paper/2308.09888/) (2023)

## Built on by (most cited first, 6 of 6)

- [Survey and Tutorial of Reinforcement Learning Methods in Process Systems Engineering](https://mlsanity.com/paper/2510.24272/) (2025)
- [Direct Regret Optimization in Bayesian Optimization](https://mlsanity.com/paper/2507.06529/) (2025)
- [BOOST: A Data-Driven Framework for the Automated Joint Selection of Kernel and Acquisition Functions in Bayesian Optimization](https://mlsanity.com/paper/2508.02332/) (2025)
- [Neural Nonmyopic Bayesian Optimization in Dynamic Cost Settings](https://mlsanity.com/paper/2601.06505/) (2026)
- [BayMOTH: Bayesian optiMizatiOn with meTa-lookahead -- a simple approacH](https://mlsanity.com/paper/2604.12005/) (2026)
- [How Many Initial Points Does Bayesian Optimization Need?](https://mlsanity.com/paper/2607.04356/) (2026)

## Abstract (arXiv)

To avoid myopic behavior, multi-step lookahead Bayesian optimization (BO) algorithms consider the sequential nature of BO and have demonstrated promising results in recent years. However, owing to the curse of dimensionality, most of these methods make significant approximations or suffer scalability issues. This paper presents a novel reinforcement learning (RL)-based framework for multi-step lookahead BO in high-dimensional black-box optimization problems. The proposed method enhances the scalability and decision-making quality of multi-step lookahead BO by efficiently solving the sequential dynamic program of the BO process in a near-optimal manner using RL. We first introduce an Attention-DeepSets encoder to represent the state of knowledge to the RL agent and subsequently propose a multi-task, fine-tuning procedure based on end-to-end (encoder-RL) on-policy learning. We evaluate the proposed method, EARL-BO (Encoder Augmented RL for BO), on synthetic benchmark functions and hyperparameter tuning problems, finding significantly improved performance compared to existing multi-step lookahead and high-dimensional BO methods.